As software and applications have become more mission-critical across the government, the need to develop and deploy new solutions and capabilities to the workforce quickly has increased. The need to rapidly develop and deploy secure applications has given rise to new approaches to application development – including DevSecOps, where security testing and infrastructure management are shifted left in the development process to increase efficiency, eliminate the need for expensive and time-consuming rework when problems are identified, and get solutions from conception to deployment in rapid time.
One of the tools or enablers of DevSecOps and this rapid deployment of new capabilities is an approach to infrastructure management called “Everything as Code.” In this approach, all infrastructure provisioning, definition, and management are done through software. This approach accelerates the provisioning and management of infrastructure, enables automation, and generally makes it easier to deploy applications quickly by allowing for resources to be scaled up and configured quickly.
But are there unintended consequences or negatives to moving to an “Everything as Code” approach? Does it lead to configuration sprawl? And can it put the provisioning and management of infrastructure into the hands of individuals that aren’t qualified or experienced enough to ensure that it’s secure?
The hosts of the ContinuousX Podcast, Rick Stewart and Michael Fitzurka, recently sat down with Checkmarx’s Peter Chestna to talk about these and other topics.
Click the “PLAY” button below to watch their conversation, or scroll down the page to read the transcript.
Transcript: ContinuousX Podcast (Season 3, Episode 4) Checkmarx’s Peter Chestna on Everything-As-Code Everywhere All at Once
Rick Stewart: Hello and welcome back to another ContinuousX podcast where we try to “Solve for X in the SDLC equation.” We’re back with Pete Chestna, North America’s Chief Information Security Officer at Checkmarx. Hello Pete.
Peter Chestna: Hello.
Michael Fitzurka: Hey Pete. We’ve seen a trend in the industry and the government space moving towards this everything-as-code approach; infrastructure, configuration, security, etc. which we feel is a very healthy DevOps mindset. But it can lead to configuration sprawl, and it does require a change in thinking, which is something we touched on last episode. Do you have any advice for our audience who’s attempting to adopt and maybe enhance their X-as-code initiatives?
Peter Chestna: Ah, great question, and first is you need to think differently about infrastructure as deployed. So, while you may still be able to use things like Qualys to look at the configuration to say, “Is this secure?” The remedy for “it’s not” is different.
So, you need to go back to that, “pets vs. cattle” talk track. Then look, you may go and log into a production instance and try to figure out what to do, but the “do part” has to be left in the code. And you go back, and you do that classic repaving. We’re gonna knock everything over and we’re gonna redo it, so you don’t have the configuration drift.
There are strong reasons to have that everything-as-code, part of which is that, so I don’t have these drifting things that are exactly similar to one another. And also, the scale at which we do these… in the old days, when I started coding, things were lovingly handcrafted. You’re looking at those pictures of all of the nice cables between them and they’re all like, oh my God, look at that. It’s so beautiful. It looks like art. We don’t do that anymore. It’s done in code.
So, the thought’s that, you don’t have that kind of time. And we’re not dealing with one or two or five, we’re dealing with hundreds or 1000s of infrastructure instances. So, you can’t go through and make the change everywhere. You can’t go unplug from A and plug into C on every single instance by hand. It will take you forever. You’re gonna get it wrong. You’ll miss a couple. Then what?!
So, this thought of having strong controls around the shift-left portion of, “can I find something that looks at the code to understand what the outcome is going to be?” That’s the shift-left of my infrastructure-as-code, or my everything-as-code.
And then I’ve got the scanners like Qualys that come in and say, “Well, is the result what I expected?” And if I don’t see that, then either I need a different tool, or that code needs to change to get that desired result. Just a different way of thinking about it, but it becomes very repeatable.
And in our last segment, as we talk about automation, especially in that CD portion of the pipeline, that we’re saying; hey, I can go plow down the infrastructure and bring it back up in a highly repeatable manner, in a very safe way, that I know is going to happen the same every time because it’s not a checklist that someone’s following and they miss a step, because someone walked into their office or the cat walked across the desk or whatever happens in those cases, or I’m under intense pressure because security incident now, or outage.
Those are things where having things in code, that say this is the way it’s going to be, and if that’s wrong then fix it here. That’s really the guidance that I have, is to embrace that – to say it gets you out of the human problems that you’re going to have, just because we’re people and we make mistakes.
Rick Stewart: Mike and I have had history with the “Brents” of the world, if you read The Phoenix Project, where you had the heroes and they are gold when they’re correct and when they’re on top of their game, but the Mack Truck Theory does apply. If that person goes away there’s chaos. So, I love the motion towards everything-as-code.
Not only does it remove the heroes, or reduce their impacts, but it also allows visibility to the entire stakeholder community, if interested, and ability to improve the process. Whereas before, you would have a configuration manager, you would have an operations person who would say your application is ready and magic occurs and it’s in production.
That type of visibility is now being opened up to all stakeholders in this DevSecOps cultural transformation, which I think is important for all parties involved.
Peter Chestna: But the one downside here is that, again, as I think about what I used to do, right, so very high level, I wrote first-party code. I used to write Windows in the SDK. So that part is now said, well, now there’s second-party code. Now I’m writing libraries that are shared. I’m bringing in open source, so I need to understand how to do that securely. Now I’ve got all that infrastructure concerns.
As an engineer, I was never trained to do that. I don’t understand how to do it securely. I was never part of the people that racked and stacked and cabled up and installed operating systems.
But if you think about containers, I’m picking operating systems. You think about infrastructure-as-code, I’m building machines. What’s the right way to harden those? We have people that have spent a career on understanding how to harden infrastructure. And now, in come developers… and it’s a good thing because it’s repeatable… but they’re going in and creating this stuff from scratch.
How do we make sure that they’re enabled to do it properly, that there is right tooling there to understand the security implications of it, and just help them get better at it. We’re stretching them in ways that I talk of, as this kind of continuous knowledge gap because every day there’s some new thing that it is “as-code.”
And now I’m gonna go do. And you look at the simple examples in SDKs. And like, oh yeah, I can do Hello World. Well, Hello World is interesting, but secure Hello World is way different. And there’s a whole lot of other things that you need to think about. This Hello World is great, but it’s a two-edged sword, that we need to help them do it securely.
Michael Fitzurka: I’ve always looked at it as sort of like moving from the Victorian era of handcrafting to the Industrial Revolution where you’re mass producing. And that’s really what we’re at, in that a lot of us old curmudgeons probably are still in that mindset of; I can craft the best sidearm possible, but then the Colt-45 comes along. And then you’ve faced the thing – the Industrial Revolution. A part of me does still miss the handcrafted – This is the perfect tool, that fits this one exact situation perfectly… that I built with my own two hands!
Peter Chestna: I was working on a product that was one of the first, if not the first, internet-driven or intranet-driven applications. We had a product that you would put into IIS, before there was IAS. And I had a DBA that worked on our application team. And we would go to him say, what should this configuration look like? And I would get questions like, well, how many spindles are there? And what speed are the disks? and like? No, no, no, you don’t understand, what I need is… a generic that solves for most of the cases. I need the 80/20 solution. You don’t get to know any of these things.
And it’s the same thing with this X-as-code stuff where its running on some other hardware and it needs to be appropriate and good enough. And you don’t have those experts that are in there tinkering and moving things and well the airflow isn’t right or whatever. You need to think about the problem differently in that mass produced world.
Michael Fitzurka: There is a goodness in a way of looking and feeling about when you create something that does continuous monitoring or continuous deployment, that takes those configurations and then validates every hour that this is exactly what you detailed it was. And if you need to change it, you just change the infrastructure part of it. There’s some craftsmanship there as well that hits the spot for the Victorian crowd.
Peter Chestna: Well, as a detective control also, if you are assured that there is no drift, then you should be able to measure no drift. If you’re in there with your Qualys-type products and saying, hey, something moved. that’s bad! Guess what? That’s another way to detect something nefarious going on in your infrastructure. If everything should look the same, then they should never look different. And that’s another control point that you can have.
Rick Stewart: Terrific. Thanks to listeners for listening to Pete’s informative information. And next we’re gonna talk about microservices. And it’s going to put this discussion on steroids when we start talking about hundreds, 1000s, 10s of 1000s, microservices running, as opposed to the monolith. So, stay tuned for our next episode with Pete. And thanks for listening to today’s episode where we try to “Solve for X in the SDLC equation.”
