Penetration Testing: Environment Selection Guide

 You’ve got a system that needs to be tested, but you’re not really certain about which environment the testing should occur in. Or, maybe you’re feeling uneasy about testing within production. Many have been in your exact same shoes in the past -- below, we’ll help assist you in making this important decision. 

Common IT Environments 

First, let’s discuss common environments. Most organizations have two or more of the following environments: 

• Production – where core business functions occur 

• Preprod /Staging – a mirror of production used to facilitate eventual deployment to production 

• Test/QA – used by developers, IT and/or QA to test new deployments 

• Dev – used by developers to rapidly test development changes 

 While production takes the crown as “most important,” each of these environments plays a critical role in your IT department. Most organizations don’t readily deploy to production without a rigorous QA testing process – untested changes could result in downtime, data loss or other or other security concerns. 

Testing Changes vs Periodic Testing 

Before we begin the debate, we should discuss the two most common forms of penetration testing: 

• Continuous Penetration Testing – when a significant change is made to a system, the changes made, and only the changes, are pen tested. 

• Scheduled Penetration Testing – periodic testing which is comprehensive of the entire system. 

Organizations that implement Continuous Penetration Testing do so to avoid introducing security risks into their production environment. Therefore, it makes sense that this testing should happen in a non-production environment. With that being said, system owners should do their best to ensure that the representative environment mirrors production as much as reasonably possible – more on that later. 

Scheduled Penetration Testing is comprehensive of the entire system, and not just recent changes. While this does result in a significant increase in effort, there are two major benefits. First, new risks can be discovered: new vulnerabilities or attack methods may be discovered since the last test. Second, these comprehensive tests are commonly required per compliance (PCI, FedRAMP.) 

Regardless of which cadence you choose for testing, one question still remains: where should your organization conduct these tests? There’s much more nuance to this question, and the answer you’ll arrive at ultimately depends on the reason you’re aiming for the penetration test.

Arguments for Non-Production Pen Testing 

Lack of Requirements 

Testing in a non-production environment may meet your requirements. For example, if you’re conducting a Scheduled Web Application Penetration Test against your product to suffice a client’s request, there may be no requirement for the environment in which the test is conducted. Provided that’s true, it’s not unreasonable to perform testing in a non-production environment.

Production Downtime 

All organizations treat production as the organization’s crown jewels, and rightly so. Many organizations have tales of the pen test gone awry, wherein a critical business function went offline. The most common mitigation to avoid production outages is largely to test outside of the production environment.  

Data Sensitivity 

Production contains real data – likely customer data. Any unauthorized disclosure of this data poses a real risk to your organization.

Arguments for Production Penetration Testing 

Mandatory Requirements 

Point blank – testing within production may be a requirement. This typically comes from three different forces: 

• Compliance – many compliance frameworks, such as PCI or FedRAMP, require pen tests to be conducted in production. 

• • PCI, this may be alleviated by your QSA performing a check to ensure that the non-prod and prod environments are sufficiently matched. 

• • FedRAMP, testing in a non-production environment can be marked as a finding on the Risk Exposure Table (RET). Your agency will determine whether or not this risk is acceptable. 

• Internal – some organizations may have security policies which dictate that testing should occur within a production environment. 

• External – your customers may have procurement policies which dictate that testing should occur within a production environment.

Production and Non-Production Mismatch 

Depending on your system’s architecture, it may be difficult to ensure that production and non-production environments are matched when it comes to security. Production environments are generally more secure. However, as we’ll see below, there are multiple scenarios in which a production environment can have a vulnerability which is not present within the non-production environment. 

Scenario 1: Configuration Differences 

Inherently, production systems will utilize different configurations from their non-production counterparts. For example, production systems shouldn’t communicate with non-production databases. There may also be certain production-specific configurations, such as security policies or features flags. These production-specific configurations cannot be tested if testing occurs in a non-production environment.

Scenario 2: Broken or Mock Functionality 

Some production systems, such as those that rely on a costly third-party API, may be broken in a non-production environment. In some cases, these third-party APIs are mocked if the API provider does not offer an inexpensive non-production API. Production-specific functionality cannot be tested if testing occurs in a non-production environment. 

Scenario 3: Non-existent Systems 

Entire systems may not exist within the non-production environment. Here are a handful of common network devices that aren’t usually present in non-production: 

• Expensive appliances 

• Storage arrays 

• Security cameras and access control systems 

• IoT devices 

• Printers, wireless access points, workstations, and laptops 

Given the impracticality of these devices to have a representative device in a non-production environment, these devices cannot normally be tested in a non-production environment. 

Still Uneasy About Testing in Production? 

If you’re stuck in a hard place where you really don’t want to perform production pen testing, but it is still considered a requirement, there are a few mitigating strategies we recommend. 

Availability concerns can mostly be eliminated via proper communication and planning: 

• Require testing to be conducted outside of an SLA
  
  
• Ensure that all teams which may be affected by the penetration test have backups of servers and databases
  
  
• Consider announcing the penetration test internally -- such as via change control procedures – to ensure that teams are notified
  
  
• Ensure that your pen test provider is aware that: 
  • All testing should immediately cease should a production host appear offline
    
    
  • Denial of Service (DoS) testing is considered out-of-scope
    
    
  • Any testing which should impede any system or network should only be conducted with specific prior approval
    
    
  • Operations which may result in excessive database entries should be earmarked with a searchable phrase such as “Pen Test Provider, LLC” within fields
    
    
  • Scans should be tailored in-nature with reasonable time delays and threads
    
• Require testing to be conducted outside of an SLA. 

Data disclosure concerns can be mitigated by: 

• Ensuring that your pen test provider is vetted by other organizations and/or your procurement department

• Ensuring that your pen test provider is under a non-disclosure agreement (NDA)

• Ensuring your pen test provider is aware that: 

• • Intentional access of customer information is entirely out-of-scope, unless it is absolutely unavoidable in proving the existence of a vulnerability

• • Should any inadvertent access of customer information occur, the pen test provider should notify your organization and sanitize obtained records

• • In some cases, specifically web applications, intentional access of customer information may be avoided by attempting to access “dummy data” specifically created for the pen test

Interested in More? 

The Penetration Testing section on our Learning Center contains a wealth of other articles discussing the nuances of penetration testing from both sides of the table. Additionally, our large penetration testing team is ready to take on your pen test needs – anything from traditional external testing, all the way to hardware and physical testing. If you’re interested in having our experts examine your systems, fill out a scoping questionnaire and we’ll reach out!