By Suzy Ahn, Head of Marketing at Nullify
Introduction
Shan (CEO and Co-Founder of Nullify) and I were lucky enough to sit down with James Berthoty, and learn more about his journey as a Security Engineer III at PagerDuty and as the creator of Latio, a library that consolidates hundreds of security tools. James shared how he started in system administration and then moved into application and cloud security, what led him to write his article “WTF is ASPM”, the impact of AI and LLMs on application security, and what he would like to see as the future utopia of appsec.
Philosophy, sys-admins, startups
Like many others, James didn’t set out to become a security engineer. But what’s unique about his story is that James didn’t even start out as a software engineer. James began his career in tech as a sys-admin, then worked as an IT operations manager. From there, he ventured into cloud security engineering and architecture where he really dove into product security and cloud native applications.
James is also currently doing his PhD in Philosophy, a passion project of his. Although on the surface his PHD is largely irrelevant to his work as a security engineer, James shows a passion for looking more critically at why things are the way they are.
Container security vs application security
Shan mentioned how the line between container security and appsec is blurring, and it’s now becoming more widely considered under the umbrella of ‘product security’. He asked James what parallels he’s seen from his experience solving cloud security problems versus application security.
Although he mostly talks about product security now, James’s main interest and background is in Kubernetes security.
“As I was doing cloud security stuff, the difference between a Kubernetes deployment and the application itself is vanishingly small. And so that's where I was getting more and more involved in the application side and then became a developer through that. Honestly, that's why a lot of what I talk about now is ASPM and product security, but my background is more like Kubernetes security. But the application security market's just really hot right now. I think that's what has made people just reach out to me, even though I'd much rather talk about Kubernetes security all day.”
James describes the difference between container and application security as more of a question of ownership.
“That's really where you would notice like hey, this JavaScript vulnerability is showing up both on the container scan and in the SCA scan. Like why is that happening? Then all of a sudden you're looking at a package JSON file. To understand what that is you have to have some basic JavaScript going in order to try to fix a container vulnerability alert.”
This then engages a key issue in application security, wherein patching vulnerabilities becomes a headache - you fix one thing that messes with another thing. What seems like a task for a junior dev ends up having to be sent to the chief architect to solve. As James puts it aptly:
“Everybody wants to act like patching is this like low brain activity because Microsoft servers are what we're used to, ‘just run the patch with Tuesday updates’, but it ends up being one of the hardest things you can do as a developer.”
The low ROI of application security
James mentioned that he saw the ROI of many appsec tools as being very low. In his experience, he implements a tool, it starts listing a bunch of vulnerabilities, and then they don’t get fixed in a timely manner for various reasons.
This is key to why James thinks people underestimate the workflow aspects of the tools they buy in appsec. He sees workflow as the fundamental difference between tools - how are you actually going to get a developer to fix something? He doesn’t see an incentive for developers to fix vulnerabilities when the numbers only go up and up.
The low ROI of appsec is problematic for obvious reasons, and patch management is an example of a task that looks simpler than it actually is. On the surface, many appsec tools produce relatively minimal gain from a business perspective.
James shared that from what he has seen, the tools available have developed radically over the last five years, with major improvements in speed, UI and UX. However, fundamentally the scanning engines themselves hadn’t changed much. What had changed was the way developers and security teams work.
Has shift-left or DevSecOps misfired or failed?
Shift-left was coined in 2001 by Larry Smith; in the 20-odd years since, there have been many attempts by many organizations to move testing to earlier in the SDLC. Skeptics say that shift-left and DevSecOps have simply picked up problems from one team and then dumped them to another. We asked James for his opinion on this.
James sees the alternative to DevSecOps, as siloed security teams blocking every major release. So, he tends to focus not on whether DevSecOps worked, but on ‘how do we make it work’?
He points out that early on, his approach like many others was about stopping new vulnerabilities, then going back and fixing all the old ones. But he soon found out that’s not how vulnerabilities work: they don’t just get created with new code, but rather with old code that isn’t being maintained. As James puts it:
“Developers want to deal with tech debt but they can't because product won't let them. Security wants to deal with vulns, but they can't because product won't let them. And QA wants to fix all the bugs, but they can't because product won't let them.”
This is why James favours tools that approach fixing vulnerabilities as campaigns, not just automating PRs.
The talent shortage in cybersecurity
We asked James to what extent he saw the talent or experience shortage in cybersecurity as a limiting factor to improving product security. Shan mentioned programs like Amazon’s Security Guardians, or Atlassian’s Security Championship, where security engineers are put on every developer team. Financially, it’s not something that many businesses can implement, but would there even be enough security engineers available to fill those roles?
James saw the issue as twofold. Firstly, organizations need to offer better incentives to developers who step up, because more responsiblity for no recognition is an unnattractive offer. Secondly, he pointed out that in the past when he did get budget to hire an appsec engineer, he couldn’t find anyone who wanted to do it. From what James has seen, software engineers don’t necessarily see security as a positive career move or as something they’re interested in.
“So that's where I think the talent shortage problem is, it’s not just a money problem the way I thought it was, but it's a more holistic. It’s an interest, plus a skill set, plus a hiring issue. And companies are really short-sighted too with how they look at hires. If you have a Java code base, all of a sudden you're trying to hire a Java security expert. If you have Go, you're trying to hire a Go security engineer."
“And then conversely, the skill sets are so varied where people will ask for a product security hire and they lean towards hiring people with red teaming experience when that's not even necessarily the right fit for trying to do like blue team appsec stuff. So I think there's a lot of problems with Appsec.”
The future of product security engineering
We asked James what he saw happening to product security engineering. Is it really in the same value stream as software engineering? Should security engineering be built into the OKRs of software engineering roles? Will product security dissolve into more distributed engineering teams?
James thinks security people do their best work when they do more abstract hunting work: investigating API endpoints and finding what is public, rather than shovelling CVEs between different teams. Not only is that more fun, but it’s more helpful. When organizations get breached, it’s usually due to something like a lack of validation on some endpoint, not a CVE on an internal server somewhere that no one looks at.
He acknowledges that there’s a fine line to be drawn. Software engineers now don’t just write Java code, but they write the Dockerfile, and the IAC, and maintain dashboards - throwing security in there may not necessarily be a good thing. However, developers usually own their own bug testing, so perhaps security is the same.
Ultimately though, James sees that product owners need to be more educated on owning their security and vulnerability backlog, as ultimately they're the ones who decide where the work goes. There’s a balancing practice between the risk that this 10 million ARR customer churns because they don't have this feature, versus the risk that the CVE gets popped. Security should be considered an essential part of shipping features in the next sprint, and security should empower product teams with the risk from a threat modeling perspective, but also a prioritization perspective.
“But in order to do that it requires like a new breed of security engineer that's not so focused on ‘everything is the end of the world apocalypse because it's going to get exploited’. It requires a more realistic business sense of it. Is this realistic? What mitigating controls can we do? Can we be thoughtful about reducing the risk monitoring?”
On why James wrote ‘WTF is ASPM’?
In ‘WTF is ASPM’, James explores Application Security Posture Management (ASPM), a new term in cybersecurity driven by Gartner. There's debate over its definition: some see it as vulnerability management, others as a holistic code-scanning solution. James advocates for a consolidated platform that covers various scans across the software development lifecycle (SDLC), including SAST, SCA, IaC, container, DAST, and CSPM, as well as clear guidance on remediation and seamless integration into workflows.
James described how he wrote the ‘WTF is ASPM’ blog in late 2023 when he attended the OWASP conference and saw many vendors were self-described as ASPMs. But they all had wildly different offerings that didn’t really fit the Gartner definition.
In James’s opinion, most vendors surface results from the same open-source scanners behind the hood anyway. Why not offer that, as well as the workflow integrations, rather than just selling a tool for your other tools? James sees this view of the market as more valuable than having another definition for what is essentially just vulnerability management.
James shared how he had seen a lot of market validation for an all-in-one scanner and vulnerability management tool.
“It's not enough to just be the world's best point solution scanner anymore.”
Scanning vs remediation
We shifted focus to remediation rather than just code scanning, and asked James his opinion on Remediation as a Service platforms. James joked that appsec startups should aim to be remediation orchestrators. He reckons there will definitely be a winner in this space - a platform that can combine the functionality of code scanners, ASPMs, and vulnerability remediation.
What James described is similar to what Shayan Shafii at ScaleVP wrote in their article “The Shifting Landscape of Application Security”. We touched on the article with James and discussed how Nullify is going down the route of autonomous product security engineering, where instead of getting product security teams to interact with dashboards and perform security work, AI workers can gain context from the codebase and find, prioritize, and fix vulnerabilities autonomously as a real security engineer would.
However, in James’s opinion, he sees AI and LLMs as more valuable in detection rather than fixing. For example, ChatGPT is doing better than many existing SAST scanners. On the other hand, from what James has seen, AI is bad at prioritization and remediation tasks as it has difficulting making assumptions. Its responses hedge everything to a point where nothing meaningful is fixed.
From our perspective, the prioritization aspect is more of a data-engineering problem, wherein the success of the tool relies on the data fed to the LLM model.
At Nullify, we use Retrieval Augmented Generation (RAG) and give it the right data on the coding environment and context, as well as other long-term memory sources such as the NVD, Nullify’s previously generated/user-accepted code fixes and plans, and more.
Nullify ingests data from your environment and builds a graph that models your asset inventory, then embeds AI agents capable of performing product security work into your codebase, messaging and ticketing platforms, that developers and security engineers or managers can interact with as if they were real security engineers.
The utopia of Product Security
To wrap up the interview, we asked James what he wants appsec to look like three to four years from now. Is it fundamentally different to how things operate now?
James shared that he sees the role of security engineering as shifting away from repetitive tasks that can be left to AI, and more towards threat modelling, architecture and threat hunting. This is high-value work that security teams can focus on if theoretically, software developers could fix all the vulnerabilities autonomously alongside the right tools that enable them.
At Nullify, we see AI as a way to augment software and security teams, enabling organizations to mature their security posture faster, without spending more on headcount.
To wrap things up
We took a lot away from our conversation with James Berthoty. It was interesting to hear his perspective from the lens of the low ROI of many appsec tools and the talent shortage in cybersecurity.
We really resonated with James's advocacy for a shift towards remediation and campaign focused appsec tools, as at Nullify we’re building an AI security engineer, not just a scanning tool.
James recognizes that major improvements to the status quo can only be done by shifting behaviours around appsec, which relies on organizations prioritizing long-term change for their developers and security teams.