Phishing Tests: What Your Provider Should Be Telling You

It's no secret: many organizations view and treat phishing as a periodic checkbox assessment. It’s often a basic email template sent to an entire organization. If someone clicks the link, they are recorded and possibly enrolled in training. While this approach can certainly check the “quarterly phishing exercise” box, you should consider demanding even more from your phishing assessment. After all, when you engage with a third-party provider, they should provide both depth and value within their specialization.

Questions for Your Consideration 

We’ve noticed a few recurring themes when organizations reach out to us for phishing assessments. As such, here are a few questions you should ask yourself to decide if a third-party provider is right for you: 

Are you prepared to exempt the campaign from your security controls? 

Understand that phishing campaigns are typically done to assess the user, not your internal phishing security controls. In essence, phishing exercises are gauging user susceptibility to a real attack, and your controls are not considered part of the assessment. However, if you are aiming to test these controls, this is more in line with a red team assessment.  

Assuming you want a phishing exercise (and not a red team), in order to prevent delays and frustration from both parties, be prepared to allowlist a campaign. This is not the same as turning off your security controls – your security controls can allow a specific policy carve out for a given source email, subject line, and/or email header. 

Are you wanting purposeful errors to be inserted in the email?

Yesteryear’s cliché of easily identified phishing emails with poor grammar is no longer prevalent in today’s age of advanced cyberthreats. LLMs can be utilized to generate professional text within a phishing email, complete with mastering the English language. Therefore, we’d strongly advise against asking anyone launching a campaign to insert obvious “tells” of a phishing campaign. 

Do you have a list of sensitive topics you would not want included in a campaign? 

Maybe you’ve got an upcoming merger. Maybe you don’t want employees to think their jobs are at risk. These types of sensitive topics are utilized by real-world attackers, as well as phishing providers. Before engaging with providers, make sure you have a list of boundaries you don’t cross. While any good provider should verify with you first, make sure to check, otherwise you may find out too late after the campaign launches. 

Will the campaign be organization-wide, or against specific individuals? 

Increasingly, organizations are eyeing targeted campaigns to identify risks to specific departments. For example, through a campaign, a financial department could be targeted due to prior issues with malicious financial transfers. Or, perhaps the C-suite has had previous issues with real attackers. In most cases, however, organizations are aiming for their entire organization or a sample. 

What indicates an individual's failure? 

This is normally driven by compliance. For some organizations, clicking on a link within an email may designate a failure. To others, a more severe infraction such as entering credentials is the failure threshold.  

Are you expecting training? 

Training can be completed using a third-party provider, however, most internal security trainings are sufficient with internal or open-box solutions. Some organizations may request a tailored training, which most providers should be happy to oblige. 

Questions for Your Provider 

We also have some questions you should ask your provider. Asking these questions of your provider will help determine if they’re right for you, as well as uncover potential snags in the assessment: 

Can I see a demo of the campaign before it launches? 

Before the email is sent to the target list, we’d recommend asking for a sample of the final email sent to a point of contact. Doing so will help your team ensure the email is allowed past filters to successfully land in a user’s inbox. Otherwise, the assessment may have been conducted to only get blocked by controls. Additionally, it never hurts to check the email for a variety of errors: disallowed pretexts, poor pretexts, obvious grammar/spelling errors, or even invalid links. 

Will your campaign capture user credentials? 

You should always double-check that your provider is on the same page with the goals of the assessment. Remember - some organizations may want a completely realistic assessment to show true impact. Others may not want user data to be captured as part of an engagement and only request click tracking. 

Will your team use an adversary-in-the-middle (AitM) proxy to retrieve session information? 

In addition to capturing credentials, usage of a reverse proxy allows for an attacker to capture session information, which may result in a multi-factor authentication (MFA) bypass. For your organization, it may be worth having this performed, as it could serve as a push to implement a more secure form of MFA.  

What happens if the campaign is flagged by a third-party system? 

Even phishing campaigns from the “good guys” can be flagged as malicious by “good” third-parties. A common suspect is web browsers – in today’s age, they are integrated with third-party phishing categorization lists. These and other third-party systems can, unfortunately, flag domains and alert users of the campaign. 

If a campaign is effectively neutralized due to a third-party system, you should double-check that your incident response team did not incorrectly flag the campaign. Additionally, the provider should re-configure and validate that the new campaign will not be flagged by a third-party system. 

Strengthen Cybersecurity Through Effective Phishing Tests 

Engaging in a phishing assessment with a third-party provider can be a valuable exercise in strengthening your organization's cybersecurity posture. By asking the right questions both internally and of your potential provider, you can ensure that the campaign is tailored to your specific needs, compliant with your policies, and provides meaningful insights into your employees' susceptibility to phishing attacks.

If you’re ready to begin talks with providers, we’ll happily volunteer to engage in open discussions with your team – fill out our brief pen test scoping questionnaire and a leader will get back to you soon!