I have heard that implementing DevSecOps along with Separation of Duties (SoD) security principles is akin to running a humidifier and a dehumidifier in the same room and at the same time. Two opposing forces working tirelessly against each other causing a localized storm in the middle. But it doesn’t have to be that way! With the help of automation, DevSecOps can bring agile management and DevOps culture together with Security in a timely fashion, all while still following SoD regulations.
The intention behind SoD is to prevent fraud and errors by disseminating tasks and privileges for business processes among multiple users. This way no one user can make changes without approval and transparency. SoD concerns are detailed in security frameworks like NIST 800-53 and ISO 27001 and in regulations like Sarbanes Oxley (SOX) and PCI DSS.
Unfortunately, this has become entrenched into a separation between development and operation teams and often manifests as the infuriating reason as to why operations won’t let development look at production, even though the application is down, and all users are affected. “Fix it now, but you can’t look at it or touch it! Now tell me what shell command you want me to run.” Those scars run deep.
In more general terms, SoD means that if you are responsible for creating something, you should not have the ability to alter or hide the reporting on that creation. For example, if you create code, you can’t avoid the testing of it or otherwise alter the operations of it. Also, if you are responsible for conducting operations on an application, you can’t independently change its capabilities or its audit logs. And from a security point of view, if you create a security control, you don’t conduct the audit of that control.
This separation prevents the bias, abuse, or fraud that is possible when you report on your own effectiveness. It is not foolproof in the face of collusion, but it is effective against single bad actors.
These controls seem reasonable, and a pro-fraud position is a difficult to defend; so why would DevOps or agile management appear to be opposed to these principles? Well, one of the underlying concepts beneath modern software development is to “break down the walls” between previously siloed teams and to operate as a single cross-functional team that defines, builds, tests and delivers small elements of functionality in a short time frame, ideally straight into production.
“…separation prevents the bias, abuse, or fraud that is possible when you report on your own effectiveness. It is not foolproof in the face of collusion, but it is effective against single bad actors.”
On the surface, that appears to be a unification of duties, making it ripe for fraud. The automatic changes straight into production aspect also raise a red flag for the SoD security conscious auditors. So, let’s scratch beneath these apparent surfaces to find out what is really happening.
While it is true that Agile management merges the development and testing teams, making all team members responsible for planning, coding and testing, it doesn’t mean that one person can plan, code and test the same functional change without independent review. Functional change requests are reviewed by the entire team and prioritized and validated by business owners.
Kanban boards, or similar tracking systems, show what requests each team member has completed and what is in progress. If the tracking is automated beyond simple pieces of paper on a board, then auditing of each ticket and overall performance monitoring is easily achieved.
To keep development and testing further separated, a proper change request ticket should include the “Definition of Done” upfront, detailing the acceptance criteria for the functionality with an Acceptance Test-Driven Development (ATDD) script written in Cucumber or otherwise explicitly defined. Ultimately, changes to the application are managed and audited by source control systems, where independent code reviews not only ensure that the code is written well but safeguard against fraud or incomplete testing.
Following this pattern, no individual developer can change the application or declare a task complete until automated tests pass, and an independent review has been completed.
Furthermore, all code changes are transparent to everyone in the source code repository and the tests are continuously run and rerun in Continuous Integration (CI) or Continuous Delivery (CD) pipelines by each developer and in each automated environment. Shared CI/CD pipelines across development and operations means that changes are more transparent to every user, making fraud harder to hide.
And by integrating testing and security scanning into the pipelines the process becomes more robust. Changes that can make it through an automated CI/CD pipeline with security gates and manual signoffs are hardly going “straight into production” and have a strong audit trail of conformance to support ATO. So, with good Agile management and automation support, the teams of development and testing have been realigned and merged, but the separation of responsibilities is still present and enforced.
An unfortunate side effect of using the same “wall breaking” analogy with a DevSecOps culture change is that it implies that these separate development, security and operational teams will likewise merge and become one, which is possible, but not required. DevSecOps is about promoting communication and sharing these concerns across all members of all teams and holding everyone responsible and not about merging all teams organizationally.
It means that development must address how their changes will affect the operational aspects of the application, such as release, deployment, and monitoring concerns, while simultaneously addressing security. And it means that operations should work with development to prepare any necessary monitoring and operational automation before a change is deployed to production.
Security and operations must also help define best practices and quality gates for release orchestration and application deployment, so that development can strive to meet them while development is occurring and not after the fact.
Either as part of the same team, or while still maintaining organizational separation, developers still develop, and operations still monitors and operates. There is just more collaboration occurring and it is occurring earlier in the process, out in the open, reducing the chances of errors and fraud.
Adopting the same source control automation strategy for operational tasks, using infrastructure as code processes along with independent review, provides all the benefits of auditing and transparency that was achieved in the development phase to operations. This infrastructure automation, often referred to as GitOps, enforces the separation of duties by only allowing infrastructure changes to be made through scripts from a source control system with peer reviewed changes.
Essentially, no one has root access except the automation system which can only be operated through user-based access controls and where each run is also tracked and audited. In this standard, if a script were written either by a developer or by an operations technician, I would argue that SoD principles are achieved when the script is independently reviewed and verified by different operational personnel.
Production access is still a major component that must be addressed for compatible DevSecOps and Separation of Duties, as making a DevSecOps team responsible for addressing production outages and system failures for the applications they create is a hallmark of its approach. Unfortunately, even read-only access to production for developers can raise the specter of SoD non-compliance. Yet no access makes taking on a support role unreasonable.
The challenge is to limit access to nonpublic or confidential data while providing access to exception logs and other information for problem analysis. If read-only access is not acceptable, operations can automate a real-time feed of “cleaned” logs and relevant information to a non-production location.
Automation helps here, as well as many tools provide monitoring capabilities with no or restricted ability to modify production. Giving developers direct access to monitoring dashboards and “cleaned” data is a happy, if challenging, compromise.
I’m not saying that DevSecOps and Separation of Duties are easy to follow, but both are compatible, and taking steps to eliminate fraud should be the proper security posture for any organization following DevSecOps. It will take additional work with planning, auditing, and reviewing, but automation will help.
Fortunately, the elements of agile management, DevOps, CI/CD pipelines and GitOps practices already contain the spirit of SoD’s intent if not every regulation. Having all developer source code and all operational infrastructure scripting visible to all personnel gives fraud no place to hide and allows everyone to improve the process.
So, on your journey of DevSecOps cultural transformation, don’t fear Separation of Duties regulations, embrace them.
