Massive cyberattacks and breaches that originated in applications, including the recent SolarWinds breach that impacted as many as ten government agencies, and the more recent Kaseya breach which may have impacted hundreds of companies, have rightfully raised questions about application security and how government agencies can ensure they’re implementing applications without vulnerabilities.
While software and application vulnerabilities may be impossible to eliminate entirely, adopting a DevSecOps approach to application development can help make applications more secure. A new generation of application security testing tools that scan code for known vulnerabilities can also ensure that government applications aren’t opening them up to attack.
But there is something more rudimentary – and potentially even more effective – that agencies should be doing – training their security people in application development, and training their application developers to develop secure applications.
This was one of many topics of discussion on a recent episode of the Continuous X Podcast, with special guest, Rusty Sides of Checkmarx. To learn more about DevSecOps, application security testing, and the importance of training on application security, watch the episode by clicking the play button, or read the transcript below:
PODCAST TRANSCRIPT:
Rick Stewart: Welcome to another episode of DLT’s ContinuousX Podcast where we attempt to “Solve for X in the SDLC Equation”. My name is Rick Stewart, DLT’s Chief Software Technologist, and joining me is my co-host and colleague, Mike Fitzurka, Senior Sales Engineer for Application Lifecycle Technologies. And as an episodic bonus, we have a special co-host for you today, for this podcast, Don Maclean. Don is DLT’s Chief Cyber Security Technologist. Glad to have you aboard Don.
Don Mclean: Thanks, Rick. I’ve never been called an episodic bonus before. Always room for a new experience. Thank you.
Mike Fitzurka: Awesome.
Rick Stewart: Awesome.
Today’s guest is Rusty Sides. He’s the Systems Engineering Manager for the U.S. public sector for Checkmarx. Rusty has over 23 years of software development, sales engineering, team management, and security consulting experience. Rusty has helped some of the largest organizations in the world in industries ranging from finance, entertainment, and Silicon Valley technology leaders in the private sector, to public sector organizations within the DoD, civilian, and intel communities to implement solutions to secure their software development lifecycles.
Rusty’s background in application security, a wide range of programming languages, software architectural design, DevSecOps experience, and public sector knowledge has been an asset to many speaking engagements at government technology and security conferences internationally. Welcome, Rusty.
Rusty Sides: Thank you, Rick. Pleasure to be here.
Rick Stewart: Terrific. And as our viewers can attest, this forum is Q&A style. And each incorrect answer will cost Rusty a significant amount of carbon credits. So, answer carefully Rusty. So, let the inquisition commence.
Don Mclean: Okay. So, Rusty, as we all know, there was a recent major hack in the United States. Many people are attributing it to the Russian hacker group UNC2452, but it’s more commonly known as the SolarWinds hack. I think we all know that there are a lot more companies involved than just SolarWinds.
But, in any event, the bad actors poisoned the well. And they did so by infecting software at its source. So that when people did what security people always recommend they do, patch; what they were actually doing was downloading infected software. Do you think that DevSecOps could prevent this kind of attack?
Rusty Sides: I think it’s a really good question. And actually, the timing of this particular hack is unfortunate, in the sense that, it got a lot of press, but I think it would have gotten more press if it hadn’t come out in the middle of the whole Trump situation that was going on at the time.
That being said, all of us in the security community, obviously, we’re very aware of it. And it’s gotten the attention of a lot of customers, especially the public sector, because every branch of the military was affected by this particular attack. Lots of civilian agencies, unfortunately, were victims of this attack, and the commercial world got hit very hard by this.
So, one of the things that I think is very important to think about with this type of attack is, we’ve always talked about nation-state kinds of attacks, where we are being targeted, not necessarily by the hacktivists, although that goes on. This is a situation where it’s organized within government agencies, and there’s still a lot of debate on this attack, as to whether it originated in Russia or if it was China going through Russian networks and we really probably will never know for sure. But that being said, it definitely was driven by a nation-state. So, this is right in the wheelhouse of what our DoD and all of the CISA type of agencies that we have are looking to protect us against, is this type of international warfare in a cyber arena.
So, back to your question on whether DevSecOps could have helped this. This is a classic, supply chain type of situation. So, supply chain always needs to be a consideration whenever you’re talking DevSecOps. You need to have tools that will go out, and not only look for vulnerabilities in the software that you’re writing but look for the vulnerabilities in any of the software that you may be bringing in, either at compile time or deployment.
Open source software is used more and more today. And I’m actually seeing that happening with our customer base, where less and less code is being written from scratch, and why reinvent the wheel, let me go grab a library that’s out there. And hey, even better, it’s free. I’m gonna take this open source library, leverage it and use it. And our adversaries are a lot smarter than they used to be. So, it’s not a matter of going out there and just throwing out malicious code on my very first commit to the repository. They become trusted.
They go out there and they put good, beneficial code out there for a while. And then, once they’ve gained the trust of the community, they say, now I’m going to stick my malicious code out there. And they’re just going to say, well, he’s written good code in the past, so let’s just take and run with it.
Rick Stewart: Right.
Rusty Sides: And unfortunately, that’s where we get in this situation. And so, kind of relating this back, and you’re right this isn’t all on SolarWinds, they just happen to be a nice juicy target. A lot of customers use SolarWinds, and they happen to have software that allows you to do a lot of things at a high privilege level, so that can also bring into play zero-trust and least privilege.
Make sure that your customers only have the rights they need for that particular function they need to do and don’t go sell the farm with absolutely every capability, full administrative access to do whatever you want, whenever you want.
So, implementing it into DevSecOps, covering source code with static analysis, covering software composition analysis, following that supply chain, and being diligent, to make sure that whatever software you’re using, you’re looking into all of the different threat vectors that exist for it.
Rick Stewart: So, what you’re saying Rusty, is people like developers and operations people, we have to deal with people like Don? Is that it?
Rusty Sides: Yes, we have to deal with people like Don. I don’t think that we can get away from that.
Rick Stewart: Grrrr!
Mike Fitzurka: Rusty, one of my concerns with DevSecOps, and with including security, is that many things get shifted-left not necessarily on to the developers, but into the development phase… but… also onto the developers! And with agencies taking a more unified, secure posture and platform; what role do you feel these mainstream developers play, and how do you recommend agencies support the developers in this role?
Rusty Sides: So, the shift-left message is one that I’ve touted for years. And I’ve been in application security, I’ve had an extensive background, but I definitely have been singing the shift-left message for a long time. Now that things have matured, I’ve actually shifted my shift-left message, and now I’m more along the lines of scan often, scan early and make your security posture, rather than just shifting it left on the developers, build security into the whole SDLC.
Make sure that the developers are properly trained. And that’s about as far shifting-left as you can get, is give them application security awareness. Teach them how to write code securely.
I’ve actually got books from college, back when I was taking my classes to become a software developer. And I can point out textbook examples where I was taught to write insecure code. Because we didn’t know what we didn’t know. Taking that situation, where developers weren’t taught secure coding. There was actually a study that was done. I think a couple years ago, it was published. And it was all about which of the major, top ten universities are teaching application security, secure coding and actually making it a required part of their computer science curriculum. And there was only one university that was identified. And that is a sad state.
Hopefully, it’s better now. I need to read and actually go do some research and see if that’s better now, but for two years ago, that’s still very, very recent, and one thing that I had always done is, make sure that application security education is in the forefront. And there’s a lot of solutions out there.
Fortunately, Checkmarx has that in our offerings as well, which is a good thing to do. Because we think, that if you solve the problem at the root, at the source, where the actual developer that’s writing the code can write it securely, and in the situation where they were inherited code, because developers often are just handed a project and said here this is your baby now. You’ve got to figure out what’s wrong with it and fix it. Oh, and it’s insecure! Go make it secure while you’re at it. So, jobs are not often out there for developers now, as much as we want secure coders. So, education is a big part of that. And then implementing into the rest of the SDLC all of the different tools that we have.
Rick Stewart: And I think you hit it on the mark with your first answer, it was the proliferation of, the influx of open source. It doesn’t mean that those application stacks that have been checked, don’t have vulnerabilities. They’re just assumed, especially people are coming out of school without the proper experience from being burnt by security issues. They don’t have that experience of the skinned knees of figuring out, that all these open source libraries do have lots of vulnerabilities.
Rusty Sides: Yeah.
Don Mclean: You know, you’re expressing concern about shifting-left. And Mike said, “Well, it’s all on the developers.” Well, I would add to this, sort of the flip side, which is that security people need to learn more about how software is developed. I know because we understand more of that internally.
Also, I know that one summer, I had to interview for someone that knew how to develop…not a developer, but someone that could just review code for security vulnerabilities. I interviewed 75 people before I even got to one person that even claimed that he could do it. Literally, I’m not making that up. It was at 75 people, who even said he could do it. And even he was like, sort of, well not really, but I guess.
It’s a rare skill on both sides of the equation. And also… I mean, Rick was joking earlier about putting up with people like me. I also had the experience of scanning code, and then we would hand the scan results to the application developers, and they’ll say, “Well, what am I supposed to do?” and we’d come in, “Well, you’re the developer, you tell me”. And “Well, you’re the security guy, you tell me”. You know, and it was sort of an impasse.
So, I think more training on both sides is really required. And so, I’ll take that on my own self.
Rusty Sides: You’re absolutely right. When it comes to the security of code, network guys used to always have to worry about infrastructure and that literally meant going into some freezing server room and spending all their time in there, with all the loud fans, and noises, and all that kind of stuff. And now all those servers are mostly in the cloud.
Everything is turning into code. Infrastructure, itself, is now infrastructure-as-code. And so, you can actually have security vulnerabilities that exist in code and you’ve never even seen the hardware.
Rick Stewart: Yeah.
Mike Fitzurka: That’s true.
Don Mclean: So, it’s shifting up the stack as well as shifting-left, right?
Rick Stewart: That’s just DevSecOps, baby! That’s what it is.
Don Mclean: All right. And by the way, Rick, did you notice when Rusty said, “You’re absolutely right”. Did you see he was talking to me?
Just pointing that out.
Rick Stewart: No, I think that’s the first carbon points taken away from Rusty, forever complimenting you, Don.
Mike Fitzurka: Our actual first point reduction.
Rusty Sides: See, that’s me, trailblazer, setting precedent.
Don Mclean: Well, just to keep you on your toes, Rusty. I’m going to shift gears over to the IoT world. You know, we’re seeing more and more IoT devices in the world in general and many of them really are not very well secured, but we’re also seeing how IoT hacking is really, in my mind, a much more dangerous prospect than many other types. It’s not just bits and bytes.
We recently saw an attack in Florida where a very amateurish hacker, compared to the SolarWinds hacker, tried to poison, literally poison the well, with the reservoir. Fortunately, they weren’t successful, but on the one hand, I’m glad it was an amateur hacker because they got caught very easily. On the other hand, it was a very unskilled hacker who came pretty close to achieving a very, very, potentially tragic objective. So, you know, I’m wondering how can DevSecOps or similar concepts, address some of these concerns around IoT security?
Rusty Sides: That’s good. That’s a very good question. Because in terms of that particular attack, and let’s just kind of start with the Florida water supply attack. That’s a SCADA system, and generally in the past, any kind of connections to a SCADA system were extremely limited. You had to physically go into the building.
And this is also a situation of, this happened to be a municipality. It’s got a very low budget for IT and infrastructure and security. It’s probably unbudgeted altogether, as part of that. So, part of this attack, was they had some remote access software that they were using. They had very outdated, unsupported versions of Windows, that they still had installed and running. Probably way behind on security patches.
More and more details of this attack start to come out, and you realize that, very poor password policy, shared passwords across everybody in the organization. They were used to people connecting through TeamViewer and just suddenly taking control of their desktop. And they’re like, it’s one of the supervisors, he’s logging in from home again because of COVID. I’m watching the mouse move around the screen. This is normal. Wait a minute! Why is he trying to? … Wait, no! You don’t need to do that to the water! … And he happened to catch it, while it was happening.
Don Mclean: Yeah.
Rusty Sides: So that stopped the attack, so to speak, but it raised awareness that there’s a lot of things that can happen here. And this happened a week before the Super Bowl, which was just right there in the Tampa area as well. So, it makes me question whether or not this particular attacker was a sophisticated attacker or really just someone that didn’t know what they were doing and started hitting buttons.
Because, if you wanted to go out there as a sophisticated attacker, and do this attack, and make it successful, they would have gone in and investigated the system enough to turn off all of the alarm triggers and everything that would have stopped it. Which the fail-safes actually did kick in. There’s no way that water was going to get distributed because it wouldn’t make it past the quality controls.
But the fact that their IT and infrastructure was taken over, brings back horror stories of all of the small municipalities, cities, hospitals, that were getting hit by ransomware not too long ago. And that was a lot more severe than this, in my book, because of the fact that ransomware attacks, they’re completely locked out of their systems, most people don’t have good, sound backup strategies to be able to just restore and keep going. And so, they’re in a situation where I either start over, or I pay this ransomware and get up and running again. And here, where I live in Alabama, Leeds, Alabama, was one of the victims of that. And they actually were like, look, our whole budget for operating this whole city government is less than the ransomware that you’re demanding.
And do we negotiate and allow them to literally get away with this crime? Or do we shut down our government and say we can’t move forward because we’ve been attacked, and we weren’t prepared for it? So, it really put everybody in a conundrum of, “How do I handle this type of situation?” So, I see this attack, almost as an awareness campaign, that here, this is how vulnerable our systems are.
And when it comes to IoT devices, those…they’re everywhere first of all. I live in a smart house. We’ve got everything from the automated thermostats to security cameras, to the robot that runs around cleaning. And these things are all made for functionality first. And get them out quick, because it’s a very fast-moving industry. And security is an afterthought. And a lot of these have default passwords, that are well known. And these users that go out and buy them, and don’t know anything about securing their device, don’t open the manual, where right there in bold print, on the first page, “Do not use the default password. Change this, first step.” They’ve got the default credentials out there, and hackers literally will go out there and just scour for: Who didn’t bother to secure their device? And what can I take over? And what havoc can I wreak?
Don Mclean: I remember a Black Hat talk where some penetration testers talked about how they were able to get into radiation detectors at a nuclear power plant. What they did was something, literally any nine-year-old could do. They got in through WiFi that uses the default password, that they Googled.
Mike Fitzurka: I am not going to sleep well tonight.
Don Mclean: Fortunately, that product’s been taken off the market and replaced. But still, it’s an indication of just how vulnerable these systems are.
Rusty Sides: Yes.
Rick Stewart: Well, speaking of that, and every one of these IoT devices, and these new attack vectors, well, there’s software running there. And I know a lot of agencies, even though there’s DevSecOps, and the movement towards locking down everything, it’s still really in its nascent stages. Each day new, instructive solutions or innovations come out.
But agencies are trying to standardize their development processes, using DevSecOps or DevOps, in shifting-left security, but they’re still some of their operations groups are kind of locking down the language choices that their software’s written in. And I think that the toothpaste is out of the tube there. There’re too many languages proliferating other than java and C/C++. Do you see any issues with operations locking down this approach and saying, “You must use java”, “You must use C”, etc.?
Rusty Sides: Yeah, the languages, both legacy and the latest and greatest language, that seems to be coming out with a new version of it every week, new frameworks to support it, it’s going to continue on. And, while the tech stacks are important, I don’t know that necessarily locking down to say you have to use a particular language, a particular tech stack, is the answer. You just have to make sure that whatever tools you choose, whenever you’re building your SDLC, your DevSecOps pipelines is covering as much as possible.
There is a multitude of open source software out there that is very niche, in the sense that if you are a… let’s say a Ruby developer… there’s a tool out there that specifically scans Ruby, does a decent job at it. But then, you’re locked in. If you have anything outside of Ruby, you’ve got to go hunting for another tool. So that’s generally where the SaaS commercial vendors are really doing well.
Checkmarx, having a SaaS tool, this is definitely an area where customers come to us. Because they want one tool that can scan all of their different languages. We have very wide language coverage, very wide framework coverage as part of that. And so, that gives them the option of knowing. Okay, now I don’t have to carry around all of these 25 different open source tools, that may or may not do a decent job. And I don’t have time to go out there and make sure that it’s still a supported community project. It’s actually backed by a commercial company. When something goes wrong, I can contact them and have support vs. just hope somebody in the open source community will look at my issue and fix it.
So, taking advantage of scanners that are out there in the open source community, and commercial scanners, really gives you the capability of making sure that all your legacy software, along with the latest and greatest, newer technologies is covered as well. And by having that holistic view, of knowing that you’ve looked at your source code and you’ve scanned it, is a good way to address that.
And I kind of add-in, along with what we were talking about before, with these breaches, especially in the case of what we talked about at the beginning of this podcast, with the SolarWinds attack. Now source code from Microsoft has actually hit the headlines, that it was stolen as part of this attack. So, part of their Azure source code is out in the wild. And Microsoft is doing their due diligence to make sure that there’s awareness, and they’re making sure vulnerabilities are addressed in their own source code but, any source code that’s out there, suddenly the hackers and the malicious users, they’re scanning it, looking for vulnerabilities. And if you’re not scanning it and knowing where you can be hacked before the hackers find it, that puts you at a disadvantage.
Rick Stewart: Right.
Rusty Sides: So you definitely want to make sure you’re not only aware of what vulnerabilities exist in your source code, but you want to go out there and mitigate those as soon as you can.
Mike Fitzurka: Well, we usually end every podcast with just a fun, little question. It’s kind of a get-to-know-you kind of thing. Although it always to me sounds like, “Be careful what you wish for”. But, if you could do anything continuously, what would it be? In theme with our ContinuousX podcast.
Rusty Sides: I would have to answer that with continuous learning. I would constantly learn. And it’s something that you have to do in this industry. Everything that’s published, everything that comes out, is already out of date. Security is a constant moving target. So, one of the things that I always advise to anybody looking to get in this type of industry is, be hungry to learn and be willing to learn for the rest of your life. Because you’re always going to have to know what happened next week, what happened, what’s going to happen, and for you to stay on top of those technologies.
You really have to take advantage of all of the different resources for learning that are out there. And actually, the viewers that are viewing this type of podcast. This is a great opportunity to hear from subject matter experts, their opinions, and their take on what’s going on in the world, in the cybersecurity world. And it’s a great way to learn and really kind of keep your skills sharp and understand what’s happening in the world.
Rick Stewart: And Don too.
Rusty Sides: And Don too. We’ll throw you in there.
Don Mclean: You stole my line, Rick.
Mike Fitzurka: You gotta be quick.
Don Mclean: Gonna say, Rick too.
Rick Stewart: Alright, awesome Rusty! That’ll do it for this episode. We thank all you viewers for your time and attention. Thanks to Don, for his thrilling support as co-host. And most importantly, thank you, Rusty, for the information you shared, as we continue to “Solve for X in the SDLC equation.”
