Devops Audit Developers With Access To Production

DevOps, like Agile development before it, accents the continuous evolving state of software development, particularly in cloud-base software. Like any technology change, there is no surprise that auditor and security professionals are challenged as the traditional separation of duties become more and more gray. As someone who oversaw product management in an Agile / SaaS development environment and now manages audits and certifications for leading edge cloud solution providers, I offer my perspective.

Agile and DevOps are similar in the sense that they both emphasize communication, collaboration, and integration among software developers and IT operations personnel. However, Agile also puts a heavy emphasis on the roles of the product owner (product manager) and end user. We also must keep in mind that Agile has also been around for several years and has the luxury of multiple documented methodologies and frameworks.

DevOps is a response to the interdependence of software development and IT operations. Its goal is to help an organization rapidly produce software products and services. DevOps has actually been in practice for a few years, although gained US prominence with its use by companies such as Google and Facebook. A good overview of the newer DevOps concept can be found on Gene Kim’s website as part of a book he, Patrick Debois, and others are writing on the topic. 

In short DevOps includes the following key characteristics:

• Use of Agile development processes
• Increased collaboration between development and IT operations personnel going as far to create the blended “DevOps” role
• Rapid and frequent releases (elevation to production), sometimes multiple times throughout the day
• Increased automation for application development and production management, including heavy use of cloud and virtualized environments
• Ability for development personnel to access production systems to troubleshoot and remediate issues that arise

So what does this mean to an auditor? Testing separation of duties is as customary to IT auditing as testing password configurations. Whether it is PCI, ISO 27001 certification, or an SSAE 16 examination, almost all assessments include some type of comparison between who has the ability to generate source code and who has the ability to promote it to production. These controls are still important to the fundamentals of IT auditing and the underlying compliance requirements still hold true, that software development controls must provide reasonable assurance against unauthorized changes to production systems.

So now what? Failed PCI assessments? Qualified SSAE 16 opinions? While it is true that separation of duties is one of the common reasons for SSAE 16 report qualifications, it is also an area that Schellman has seen emerging trends towards heavy use of compensating controls for both SSAE 16 examinations and PCI validation. The following are some examples:

• Automated and traceable authorizations for promotion of code to production
• Role-based access controls that acknowledge when DevOps personnel have access to production systems and document the specific use cases
• Encryption and logical access controls which essentially “lock-out” the cloud provider from the data of its tenant customers
• File integrity monitoring (and alerting) on changes to production code versus the traditional focus on critical operating system executable
• File access monitoring on the source code itself with appropriate alerting
• Extensive logging and daily, if not real-time, log review of the above data sources

It is important to note that from a monitoring perspective, changes to files must create notifications that span across functional groups. In other words, a change alert must be sent not only to the DevOps team but to management and/or an independent security or compliance function within the company. From a process perspective, the increased communication and collaboration inherently creates an environment whereby an unscheduled or unauthorized change is more likely to be noticed. It goes without saying that there should be a documented follow-up and response process for any such notifications. Furthermore, daily deployments, coupled with necessary roll-back procedures allow the potential impact of an unauthorized change to be reduced.

Several people may argue that some of these compensating controls would be detective in nature and not preventative. But what is the alternative? Fail the control, qualify the opinion? More importantly, was your organization really that much more secure under the traditional waterfall model?

As auditors, we need to do what we are supposed to do – understand the processes, evaluate the controls, identify the potential impacts / risks, and provide an independent report. Then the users of the reports can make an intelligent decision as to whether or not they were better off with a traditional development methodology. From my vantage point, DevOps and Agile are here and will continue to be so until a new approach that bends the rules even more is introduced.