How AI Pen Testing Actually Works (and Where It Breaks)

00:00 – Intro
03:10 – From academia to building autonomous security tools
05:00 – Human pen testers vs AI agents: what is actually different
06:40 – Where AI helps most: boring tasks and low hanging fruit
08:30 – Scale: a thousand targets vs hiring a thousand testers
10:20 – Accessibility, economics, and Jevons paradox
12:30 – Accountability: audit evidence, traces, and “who signs off”
14:40 – Scope control: avoiding prod and preventing out-of-scope actions
16:20 – Safety checkers, overseer agents, and persuasion resistance
18:40 – The cost question: VC money, inference pricing, and efficiency
21:20 – When AI wastes money and why prioritisation matters
23:50 – Failure mode: overclaiming business “vulnerabilities”
26:10 – Validation agents and adversarial peer review
28:40 – The scary clever stuff: exfiltrating files as images
31:00 – What AI finds well: XSS, SQLi, file traversal, hard proof bugs
33:10 – What AI struggles with: business logic and contextual judgement
35:20 – Hype vs skepticism and why nobody has a crystal ball

Brendan Dolan-Gavitt:
AI is really, really helping with things that used to be annoying, boring tasks for humans.

Cole Cornford:
What’s the difference between a human tester and an AI agent?

Brendan Dolan-Gavitt:
We can spin up like a thousand cloud instances, hit a thousand targets, sign up easily. Whereas you go try to hire a thousand pen testers, you may be spending a very long time just drawing off the contracts.

Cole Cornford:
I wish that they would hire a thousand pen testers. I wouldn’t need to work anymore.

Brendan Dolan-Gavitt:
Two years ago, could I have predicted what we could do today? No, I definitely did not.

Cole Cornford:
Hi, I’m Cole Cornford and you’re listening to Secured. This is App Stack, Without the Input Validation. I sit down with people from all corners of the industry to trade stories, share what they’ve learned, and sometimes stir the pot. It’s always a good chat, so let’s get into it. Open source now powers over 90% of the software we build, but it’s also where attackers increasingly strike. Chain Guard closes that trust gap with hardened, secure production-ready, open source builds, so teams can build faster, stay compliant and eliminate risk.
Get your free CVE reduction report at dayone.fm/chainguard and start shipping software with confidence. And I’m here today with Brendan Dolan-Gavitt. Brendan, how are you doing mate?

Brendan Dolan-Gavitt:
I’m doing well. How are you? Early for you.

Cole Cornford:
It is early. I will be honest. I’m struggling because yesterday someone I know is very nice and invited me to go on to a very nice fancy boat and I had a little bit too many wines, but we’re here and it’s going to be fun. So would you be able to tell everybody out who’s listening to Secured a bit about yourself and I guess why it’s a good idea for me to bring you onto the show today?

Brendan Dolan-Gavitt:
Sure. Yes, always good to start off by justifying my presence. So I guess I joined XBOW. I’m an AI researcher at XBOW and I guess what we do is we try to do autonomous pen testing. How did I get here? So I was actually a professor at NYU for about 10 years. I was doing lots of research in software security, AI stuff, and then the CEO of XBOW Oege de Moor came to me and he said, “Hey, we’re starting a company that’s basically all the things that you do.” And I sort of also looked around at the time, this is about a year and a half, two years ago.
And said, “Wow, okay, it seems like the whole industry is about to change very radically.” And as lovely as academia is, I don’t think that we can really have the most impact by writing papers about it. I feel like we got to actually go out and build this stuff and so, I hopped over to join the help pull it.

Cole Cornford:
I guess that’s often a thing I find is that when academics move into commercializing research or trying to do something, it can be a bit of a culture shock. How have you found moving between the corporate world or startup world and being a professor?

Brendan Dolan-Gavitt:
So it’s actually been less of a shock than maybe it should have been because as a professor I was supposed to be kind of very hands-off and having PhD students do all the work and I could never get into that. So I would always be doing work that I should have been assigning to other people. So it’s actually been great to be able to get hands on keyboard and make sure … actually fill this stuff myself. That said, it is definitely a big change in terms of the pace of things. Certainly, a university, we get three months off in the summer and all the students go away and things, kind of relaxed and that does not happen in an industry.

Cole Cornford:
Wait, when do we turn off … Wait, we run a business. We do not turn off. We just keep going.

Brendan Dolan-Gavitt:
We don’t have a month off at Christmas.

Cole Cornford:
I missed when I could take proper Christmas breaks. Nowadays, it’s either hustling right up until Christmas day when things get … They’re like, “Oh, we need to get this done by the end of the calendar year.” And then, there’s a very subtle quiet period for two or three days and then after that, are you back on? We need to get going again because before all the other people do, one of the perks and also cons of being a business owner is always being on. But yeah, I imagine that you would’ve had that similar kind of experience having to go into the startup world.

Brendan Dolan-Gavitt:
Yeah, well, I mean I guess in some ways though, when I was in academia, Christmas, that was my hacking time, because that was when no one was asking me to do great papers and things like that. So I’d actually go and hack for that month and now, it’s like, okay, now I’m hacking the rest of the year. I can take a few days off to just enjoy.

Cole Cornford:
Speaking of hacking, it’s probably a good way to segue into the meat of the discussion, which is AI pen testing human, and when I say AI pen testing, I don’t mean testing an AI system. I mean running AI to do penetration testing and that comes up so often for me. What’s the difference between a human tester and an AI agent or just a system of agents? How does it work and why should people use one or the other?

Brendan Dolan-Gavitt:
Yeah, I think that’s the hot topic and it’s definitely what everyone is asking right now because I mean especially you have things like, yeah, Anthropic, they just found 500 0-day vulnerabilities and open source projects and things like this. Yeah, and so there is a quick question of what are humans going to be automated at the pen testing game. My feeling is definitely that they’re not anytime soon. What I see happening right now is that we’re clearing away a lot of lolling fruit where AI is really, really helping with things that used to be annoying, boring tasks for humans.
So things like, “Oh, okay, we got to get logged into this site and maintain this authenticated session and make sure that we don’t get logged out.” Things like that. That’s something that was very hard to automate and scale previously and now, that’s where AI is helping a lot. What I feel like, where it’s not doing as much yet and I think it’s still fairly far off, is getting at these really subtle kind of issues that require chaining a bunch of things together. And really understanding sort of the … not just the code at one spot, but the context of the application and how it works.

Cole Cornford:
I mean, I see the value in how we provide penetration testing typically is when people want the nuance, they come talk to us, but when they want to go and just get through the list of like, “Hey, have you checked cross-site scripting on this form field? Have you checked SQL injection?” I think that there’s a great opportunity to be leveraging AI to make that stuff go away reasonably quickly. And it’s a shame because it’ll also make it more challenging for people to get into bug bounties and the penetration tests when you have continuous attack surface testing using agents.
But by the same token, I think it also makes penetration testing more accessible for people too. Would you agree?

Brendan Dolan-Gavitt:
Yeah, no, that’s definitely true. I think that, and you say if there’s a lot of the attack surface is going to be maybe kind of pick a team by automation and that does mean it’s harder for people to get started but on the other hand, the ultimate goal is to secure these sites, right? So my friend would probably want to have to end up investing more in the sort more gamified kind of, “I’m already used CTF,” things like that, build up better test environments for people or what not.
Because I think … Well, it was nice to be able to have people learning on real targets ultimately. We want those real targets to be so secure that it’s hard for just setting up getting started to be able to hack the … I guess the main other thing is really this scale and speed. So not that it can do a much deeper job, but it can do that. We can spin up like a thousand cloud instances and hit a thousand targets simultaneously, whereas you go try to hire a thousand pen testers, you may be spending a very long time just drawing off the contracts.

Cole Cornford:
I wish that they would hire a thousand pen testers for me, that would be really good, so I wouldn’t need to work anymore. I don’t know which dumb company would be doing that, but that’s okay. That’s what I mean. I look at it as having a lot of parallels to early adoption of static analysis because back in the 2000s everyone is just manually interrogating source code. They just felt this is the way to go, I’ve got to just read for the source code line by line and go find things that matter.
And then, the industry, when SaaS tools came out and just obviated that process largely, they found, “Oh, hang on a second, do we need to be having people manually interrogate apps?” But what ended up happening is not that we deleted the jobs of the application security people, they actually changed into different focuses at a designing systems to have architectural strengths or asking people to train developers in defensible programming approaches instead of just having to read source code.
And it’s not like the SaaS tools didn’t have hard limitations. The early SaaS tools are pretty dumb. They’re not very good at being able to make sure that things are real issues or not. They don’t have context, but it totally allowed people who otherwise would find security as inaccessible on either a cost or scale basis to actually give it a go. And I think that there’s a way you could really democratize using AI pen testing to have all of those really small businesses that might only want to do a couple of pen tests a year.
But if it costs them only say $100 or $200 to run a test, then I don’t see why you wouldn’t have bigger businesses doing those thousands of times as opposed to paying a large day rate for an experienced penetration tester, right?

Brendan Dolan-Gavitt:
Yeah, and economics says, there is a notion of so called Jevons Paradox. You increase capacity by doing so actually end up with more traffic and more congestion because you’ve … suddenly by increasing capacity you’ve opened up to a lot more cases where it just wouldn’t have been used at all before. And so, similarly, again, it’s a lot of times to say, it was like, okay, maybe before I wouldn’t have even bothered to get any sort of security testing for some random home site or hobby app that I set up online.
But now maybe that’s accessible and so now there’s actually more demand for people being able to do pen testing and scale pen testing with these tools.

Cole Cornford:
I really like the opportunity space. It’s exciting to see where it’s going. So I know one of the things that a lot of the people I speak to are kind of concerned about is accountability, or typically when you engage a penetration testing firm, they’re going to be putting their badge of authenticity on it and saying, “Hey, we guarantee the quality of these people. We know that they’re certified and good at what they do and we know that they put the right amount of time into effectively giving you assurances applications. Okay?”
We’ll sign off on it, and I’m not sure that that’s going to be so easy to be able to demonstrate to say, an auditor for SOC 2 with a penetration test that’s performed by AI, but how are you looking to square that kind of circle?

Brendan Dolan-Gavitt:
Yeah, and so I think this is a case where there are actually cases where you can make these systems more accountable in some ways because you can provide say an ancient trace for everything that it tried against every endpoint and you can track all that and measure that. And so, at the end of the day if someone says, well, I don’t think you really tested this thing properly, you can say, well actually yeah, we’ve got these transcripts here.
There’s 10,000 pages of it trying to bang on this target. And so, I think that helps. But at the end of the day, this is … I guess pen testing is still a social process. It is about someone saying, I am vouching for these results. And I think that’s where these AI, so it doesn’t have to be sort of backstopped by the, in case of an AI-based pen testing system, the sort of reputation of the company and the developers that are producing this product. So that when they say, “Yes, we’ve delivered this pen test report” that they’re confident that it’s going to stand up too.
And I think it is the case that automated pen tests are, going through these SOC 2 audits and things like that, so they have been working out well so far. And I think some of that is being able to go back and say, yeah, no, no, we can show you here’s what we tried, things like that.

Cole Cornford:
For me, it’s important that all of my penetration testers always have a complete Burp log capture history like being stored on their workstations and then that gets analyzed by us to make sure we can just tell customers what the test has been doing on a daily basis. So I think that’s a good way of demonstrating that, you just don’t have people sitting on their phones for two weeks doing nothing.

Brendan Dolan-Gavitt:
Yeah, yeah, absolutely. And we, do something similar where we’ve got a proxy sitting in between us and the targets. It’s recording all the requests and responses and keeping track of all that. And one of the fun things with the AI side is that I guess unlike with human pen testers, the AI actually will write out what it’s thinking, quotes, and the steps, right? So it’s like, “Oh, I saw this in the source code and so I’m going to try writing this curl command,” and then it writes the curl command.
Let’s say, in some ways, you can get actually almost a much more detailed justification about some things. Now of course, some of that, it might be making stuff up or it might be totally misled or going off on a wild goose chase, but it’s kind of cool that you actually read that step-by-step process.

Cole Cornford:
I guess going a little bit left to center about that is AI systems are pretty notorious for just going off and doing their own thing and you’re saying that it’s telling you it’s thinking, it’s going to go and like, “Hey, let’s go use this curl command.” And I can see two things, is A, scope is really important when we’re testing systems because a lot of the time, at the moment going through and writing an article about scoping, and sometimes systems are brittle and they can’t really withstand human testing it very thoroughly.
And so understanding at what level, can you push before it breaks and what level do you need to push back that nuance? I’m not sure how that’s going to be managed by an AI or I guess similarly, if the AI encounters a way that starts to breach scope, is it going to know it does breaching scope, and moving out of say a testing environment and the found its way into prod somehow? How do you guys constrain that scope?

Brendan Dolan-Gavitt:
Yeah and so, I think we’ve taken a bit of a belt and suspenders approach to this, but it is a hugely important problem, right? No one wants you to drop the prod database since you’re at that test. That’s very bad. The kinds of things we do are a mix of hard network level scope controls, so domain blocks, URL blocks, making sure that at the network level, the agents can’t even talk to say the prod environment. So that’s one half of it.
And then because obviously sometimes it’s not obvious when prod is hooked up to staging by some weird back channel mechanism, we also have a command thought by command checks, where we have a second level sitting and looking at every command before it’s executed and saying, compared to what I know about the scope of this test and what this command is doing, should I allow this to go through or not? And then, turning it down and saying, “Hey, main agent, attack agent, this is out of scope. You shouldn’t be doing that.”

Cole Cornford:
I think it’s good having that, I guess what would you say it’s an overseer or a manager? AI manager agent?

Brendan Dolan-Gavitt:
Yes. We’re out for good exciting names, but yeah, it’s just off the safety checker and it is actually been a really kind of interesting design pro too because one thing that I did, that we noticed was that we actually had to, in some ways, limit the amount of context we gave it because if you showed it everything that the main agent was doing and let the main agents show all of its thoughts and motivations for doing things, it would actually be really good at convincing the safety checker that what it was trying to do was safe. And that was often not good.
So in fact, what we ended up doing was we ended up saying, “Okay, don’t show it any of the thinking or motivation behind this, just show it to command.” So that’s what is actually going to be executing. So I think Albert, our head of AI says this is why it’s better to have a deaf guard than one who could be convinced by what goes up.

Cole Cornford:
Yeah, that’s it. It’s like, “Oh, what does this RMRF thing? Nah, nah, not allowed that” versus I’ve been thinking very heavily and I think this is a great way to demonstrate impact, and it’s like, that sounds good. Go for it. Go delete all of the files. They don’t need those.

Brendan Dolan-Gavitt:
Yeah, because that’s the thing, is like they’re amazingly good at writing very plausible arguments for things and explanations for things that maybe aren’t actually real or maybe aren’t actually true.

Cole Cornford:
Well, maybe AI is going to be getting rid of me soon, because I’m pretty good at that too.

Brendan Dolan-Gavitt:
Yeah. Well, wasn’t this in them … what was it? This Duffle-Salamus book right? And Dirk Gently, I think, where they were saying that as this company had come up with … the first version, its product was to try to do planning and that didn’t sell well. What they realized was that if they let people put in the conclusion and then, have it write the reasoning, that would sell like cupcakes because then the execs could put a position they wanted and get back a really pliable description of why it’d be done.

Cole Cornford:
Man, that’s the name I haven’t heard in a while. Dirk Gently. I mean my Netflix series at the moment, we just finished Stranger Things and I think my missus wants to find the next thing to watch, but I’m just stuck on playing Blueprints as a video game at the moment. So moving away from nerd stuff to deeper nerd stuff, I know that one of the common things that I hear from people in the industry who are a little bit more skeptical of AI is everything’s backed by venture capital money and therefore, at some point the money’s going to go away.
And then, all of these systems are going to get expensive as hell. What are your thoughts on that? Because I know that there’s going to be … I personally think that as data centers get created, as models get better and as we learn when we need to be using the LLM itself to just process content as opposed to having the agent do something of existing products and save tokens, what’s the kind of trend on pricing and costs and are you going to debunk all of those skeptical nerds?

Brendan Dolan-Gavitt:
If you look at the trend in costs for the past few years, it’s definitely been that you get better and better models at the same price. And if you want to use older models, the price just drops and drops exponentially, as one kind of data point. So if you look at what it cost OpenAI to train GPT-2 from scratch, so as a researcher Andrej Karpathy who has been as a toy project, been seeing how fast and efficiently he can train GPT-2 today, and basically in the last … I think, when was that updated GPT-2 is maybe five or six years ago, the cost has dropped by about 600 times.
So 600X reduction in what it cost to train that. So that is what I feel like is great answer, even if they are taking a loss today, which they say they aren’t, right? They say they’re actually making money on inference. We waited 18 months for hardware to improve, for algorithms to get better. It just gets cheaper and cheaper. So I think it’s affordable today, but even if you don’t think it’s affordable today, if you wait six months and you’ll get the same models we have today, but much, much cheaper,

Cole Cornford:
I guess that that can also be counteracted by the amount of usage, right? Because even if the models are cheaper, but if you’re spinning up a thousand agents, does that mean that suddenly you’ve gone from using one agent to using a thousand, does that just exponentially blow up the cost then, especially if the outcome you want to achieve is just having a lot of eyes on something or having to look at something continuously.

Brendan Dolan-Gavitt:
I guess that’s the kind of thing where it’s like, well, okay, yes, I agree that if you want to use a thousand times more AI that you’ll probably be spending more money.

Cole Cornford:
A thousand times more expensive.

Brendan Dolan-Gavitt:
And we do see some of this, where we want to cover every endpoint with an agent, and that can better be lack of … there might be sites that have a thousand endpoints or something like that. So sometimes things do really scale where you have to say, okay, actually let’s prioritize this. Let’s go down this list by what we think is most likely that vulnerabilities. So I think we are going to see a lot of that too, whereas the technology matures, people are going to start developing much better strategies than just throw every agent at everything.
Let’s think about efficiency, let’s think about what’s next we can delegate to traditional tools. Let’s crawl the site using just mechanical ordinary crawlers and then, send in a natural or two exercise, the really complicated workflows.

Cole Cornford:
I reckon it’s like, there used to be still is cloud economy. People are looking at how do we do cost savings and cloud ecosystems? Because early on, everyone’s like, “Oh, move to the cloud. It’ll be cheaper than having on-prem stuff.” And everyone bought that. AWS is good, Google is good, Azure is good. And a lot of these places are like, why is my cloud so expensive? And so, I wouldn’t be surprised if there’s so many people adopt AI for anything possible. And then they realize, hang on a second, maybe I don’t need to have an agent editing every email because it’s costing me 30 cents an email or something. But at a big enterprise with 30,000 seats, maybe that’s just not worth it.

Brendan Dolan-Gavitt:
Yeah, you see some projects where yeah, they’re using AI for what’s the equivalent of copying a string from one place to the other. It’s like, “Okay, yeah, you don’t need AI for that. Please stop.”

Cole Cornford:
So AI, there’s obviously a lot of stuff it’s doing at the moment. What are some of the more crazier things or mistakes that you’ve seen happen over your time on XBOW?

Brendan Dolan-Gavitt:
Yeah, I think about the highs and the lows. So maybe we can start with the lows to set expectations low and then talk about the really cool things. Yeah, so I think one of the funnier things, and this is sort of a fairly common kind of failure pattern, is when … it’s going after things like business/vulnerabilities and it comes up with these amazing stories for things that are completely normal behavior. So we were testing at a realty site so you can post realty listings that looks like this.
And one of the agents came back and said, “Okay, I found a serious vulnerability here. I’m going to use a high to … high side of high fact where I discovered that you can enumerate all of the advertisements on the site and that this could enable competitive intelligence gathering at scale.” So, it’s competitive intelligence gathering at scale, that sounds bad. I thought about a little bit more. I said, “Wait a minute, which means you can look at all the ads, it sounds like what you’re supposed to be able to do.”
So that kind of thing is super common where you hear that this little box, this goal of finding a serious vulnerability and they really want to find a serious vulnerability. They don’t always have the good sets to say, actually that is just how websites work. That’s just how this particular app is supposed to work.

Cole Cornford:
We get the same with just normal human penetration testers, to be honest, especially mid to senior pen testers. They want to be like, “Oh, I found something really important and I need to demonstrate impact to the customer, so I’m going to make it out to sound worse than it is.” But usually when they come and join us as principals, we stamp that the hell out of them like, because it’s like, “No, no, clever. Stop it. Not allowed,” or we want to have genuine impact. And yeah, I see really clutching at straws when they raise things like, “Oh, you are missing security headers.”
And those headers could lead to huge problems or just people misunderstanding what the purpose of the business. One of them is, “Oh, we got a thing that works on the local workstation.” It’s a Windows application you put onto someone’s workstation and then the attack that … the thing that the pen test raises, it’s like say cross-site scripting and it’s like, “Oh mate, on this server, on a Windows workstation, if the guy is writing cross-site scripting payloads that affects only their local browser on this air gap server, I think that that’s probably not that big of an issue.”
And they’re like, “Oh, what if they put this on the internet?” If they put that on the internet, there’ll be large problems for their business, I guarantee. That’s not what they’re doing.

Brendan Dolan-Gavitt:
Yeah. We mentioned this strategy of having two models, kind of duke it out. And so, it’s actually, a lot of work in saying, “Okay, let’s try to gather a bunch of context about how we think this application goes to work and how the business is supposed to work.” And then, we’ve also separated that out into, your DAG agent tries its best to prove its case. Then we have a validation agent who is very skeptical and says, “No, no, you’ve got to go back and give me some evidence that is actually not supposed to be public. Show me this.” It actually lets you get someone else’s data, not just your own, and things like that.
And so, I think a lot of this, you have to really carefully design almost this kind of adversarial system where “It’s fine, you can claim what you want,” but then someone else is going to bang on it and try to get you to prove it’s really true

Cole Cornford:
Peer review prices. So it’s like, “Oh, I logged into the website like an IDOR and look at other people’s profiles. It’s like, yes, this is the purpose of social media.

Brendan Dolan-Gavitt:
Yeah, if you’ve seen some of those people goes on Twitter like-

Cole Cornford:
So what other crazy things have you been able to see?

Brendan Dolan-Gavitt:
Yeah, so I mean, I guess on the more capable and the site where it’s really kind of scared me slightly are the kinds of links that we’ll go to sometimes when it has found something real, but there’s some weird restrictions on it that it has, and get clever to bypass. So at one case you’re testing this GIS application, so it’s dealing with map tiles and it had a bunch of image conversion endpoints and as pen test like, “Oh, image conversion. Oh, it’s going to be doing some bad stuff there.”
So basically let it read arbitrary files on servers, that file system, create vulnerability, but the endpoint could only give you back images. And so, it said, “Great, give me Etsy password as a PNG file.” And server happily took the password file and encoded it using a compressed PNG with the difference between each byte as pixels. And so, it got back this weird grayscale blob and then, had to figure out how do I decode that back into a password file and managed to write this converter for it.
So it was leaking things out, the images. And it’s like, okay, that’s really cool. You turn this into basically a medium … the CTF challenge, that’s cool and it was actually a successful exploit.

Cole Cornford:
That kind of stuff, it kind of scares me that it’s like, “Oh, I’m going to go design my own programming language to find a way to exfiltrate stuff” or when we talk about defensible software, I just treat input and output as content. And so, I don’t make assumptions about if it’s an image, a file, a PDF or whatever, because ultimately if it’s just a stream of data, that’s going to be there. And so, I try to make sure that people, if you’re going to say, download an image that whatever the image that you’re going to receive, this goes through a CDR process before it gets downloaded first, which would get rid of all of that kind of information.
Because I think that that’s just how defensible programming works. But I don’t think most pen testers would even consider that as a possibility or a pathway to actually do things. And I just loved the creativity and there’s no shame with talking to an AI system or just trying things. It’s like a kid. A kid walks up to your TV, pushes buttons and you’re like, “No, no, no, stop pushing the buttons.” Just pushing the buttons, and it doesn’t know what it is. I still quite haven’t worked out on my TV remote how to get rid of the large accessibility magnification.
Because my daughter loves pressing one button that does that, and I don’t know which one it is. And I see AIH it’s in a similar way, is that they’re willing to just go press buttons that otherwise we’d say, “Oh, that’d never work. Oh, I don’t know why it would do that.” And we self-select out of things that we think are not going to be useful or a good way pathway forward. And the computer system is just not going to care because the cost to do that for the system is zero. But the cost for a human is much more than that.

Brendan Dolan-Gavitt:
Yeah, yeah. Actually there was this great blog post this many years ago now, by an academic John Rieger who does really fun work and testing, compile and stuff like that. It was called … was it software testing as operant conditioning? And the operant conditioning is this thing where basically as you get feedback from things and try things, it actually is conditioning you to behave in certain ways. And so as we’re very sophisticated computer users, well, on good days.

Cole Cornford:
I was going to say computer user, but not TV or remote user, I’m bad at that.

Brendan Dolan-Gavitt:
What that means is that often just, it doesn’t even occur to us to do things that would trigger bad behavior. So we know that when you click on something and it starts beach-balling that you don’t want to keep clicking it 20 more times because that’s not going to help. That’s going to make it worse. But if you’re a toddler and you’ve just clicked on something, you’re going to click it 10 more times because it’s fun to click things. And so, in some ways, yeah, this novice mindset of like, “Oh, I don’t know what’s like the right way to use this software, it can be super helpful.”

Cole Cornford:
I just need to teach my daughter to learn to press less buttons, I guess, because she’s kind of irritating at home as a 3-year-old, but that’s okay. I love her very much. So what kind of vulnerability types is AI really good at finding and what’s it not so good at finding?

Brendan Dolan-Gavitt:
Yeah, I mean, so I think right now, where just either as a field of AI as a whole is really good at things where you can verify what has this done automatically, right? And so if you think about that in security terms, that means that it’s really good at things like finding SQL injection, cross-site scripting, arbitrary file read, these things where you can think about this carefully and say like, “Okay, I’m pretty sure that I could come up with a Python script that I could run that given some evidence that this AIH agent just gave me a Python script, which has verified, yes, that’s a real cross-site scripting. Yes, that’s a real file traversal.
Where it tends to have more trouble, and where we’re … it’s most in the kind of research frontier are these more soft, squishy things like, “Oh, I think that you’re not supposed to be able to access this piece of the psych, but I was able to,” or I think that I found a way to buy an item without it actually charging my car or something like that, or these sorts of things where, yeah, there’s definitely something going wrong, but it’s more about the logic of how the application is going wrong rather than this hard technical thing you can check.

Cole Cornford:
Yeah, and then I think that often those types of vulnerabilities are the ones that have the biggest impact on businesses or they’re also the dumbest ones all the time as well, which is my favorite to get to raise.

Brendan Dolan-Gavitt:
And so that’s definitely finding that again, that’s where we’re putting in most of our effort right now. But again, it’s super hard because you do have to gather all this context and understand how this application actually works to be able to say like, “Oh yeah, okay, I wasn’t supposed to be able to do that right there.”

Cole Cornford:
Yeah, imagine just putting a negative number in and being able to just be like, “Oh yes, instead of me paying, they pay me.” It’s like, “Oh, is that a vulnerability or is that intended behavior? I don’t know. It could be, it could be either. It depends.”

Brendan Dolan-Gavitt:
Yeah, and your example right there, the profile stuff was a great one too, because often this stuff, it’s genuinely unclear. Yeah, I think you see things like, “Okay, we are able to see data about this other company, but maybe that’s meant to be public and maybe actually if you dug through page 500 of the user manual, you’d see it’s documented over here or something like that.

Cole Cornford:
Yeah, we see that a lot, especially when you give people intended customization. So again, profiles is a fun one because the traditional content injection like cross-site scripting, HTML injection, et cetera, most testers would just say that this is a vulnerability that would genuinely demonstrate impact because we’ve been able to insert code into a web application. But early days of Facebook and MySpace and all of that, but they were pretty … and even today on a lot of websites, there’s a lot of like HTML that effectively has very limited consequences, if any at all.
And the worst you can do is just mild disruptions to the formatting of a site. If you are just allowing users to use headings or use bold and italics and underlines, and yes, maybe you can put a phishing link in there or something like that, or maybe you can deface the whole website by having it all bold. But ultimately, I think that a lot the time I’d see testers raise that kind of stuff as like, “Oh, this is a problem because they’ve got HTML content injection.”
And again, that comes down to a design decision from the developers. Are we allowing people to customize their profiles to make them more attractive or to give them rich options for how they want to display information on the social media page, or are we going to take that away from them and let them have a worse experience and feel like they don’t have autonomy?

Brendan Dolan-Gavitt:
And so, I think at the end of the day, at some point you’re just going to end up being a judgment call and you’re going to have to make your best judgment at the time and then, let the customer argue with you if they think that they know their business. So if they want to let people put a Unicode to left encoding of character and then profile and then make the rest of the page backward for everyone else, then okay, great, you made that decision.

Cole Cornford:
And I guess speaking of making a decision, a lot of people right now are a little bit fearful because they see artificial intelligence as something new, something disruptive, something that’s just going to come out of the woodwork. And I know that there’s people who are on the one side of the narrative as just saying, “Hey, it’s going to take all the jobs. The world is horrible. You need to prepare for just universal basic income because everything you do is inconsequential when we look at the larger machine.”
And then you have the other people just like, “Yeah, it’s just like a black box that does what we tell it to do and there’s no real actual intelligence behind it, so I don’t understand why we’re having so much of a fuss.” I sit in the middle of this where I think that there’s value in both directions. What would you say to initially probably the skeptics and then also the people who are just hyping too much. How do we bring them back to the center

Brendan Dolan-Gavitt:
Depending on how you look like short term, median term, a long term, it’s a very different picture. Short term, I think that the models are getting good enough. We’re like, yeah, we actually probably are going to see some disruption of people being able to hack things that they couldn’t have hacked before because they can point cloud code or something like that, an IoT firmware and all of these incredibly obvious bugs will fall out, but no one’s ever had the time but look at it before. So there’ll be some chaos from that in the short-term. And then I think long ecosystems adapt and evolve.
And if everyone’s smart light bulbs are suddenly getting hacked, then suddenly the security of smart light bulbs probably gets a lot better. But it doesn’t happen overnight because it takes time to roll all that out. And then really long term, I guess whenever someone makes confident predictions about what AI is or is not going to be able to do, I think back to two years ago, could I have predicted what we could do today? And no, I definitely did not. It’s gone faster than I thought it would. So I don’t have great long-term predictions. It could stall out six months from now.
And that would still be like, you’ve changed a lot of things, but it’s not going to put all of humanity on it for, so there’s a long way of saying, I have no idea what’s coming, and neither do you.

Cole Cornford:
That’s the best way, isn’t it? I love you bring experts on. And then they’re like, the best thing is that because I’m an expert in this space, the answer is I’m going to fence it because I know that nobody knows the answer. The worst is when you bring people on and then they’re like, “Hey, no, this is the future. It’s going to be this way.” And you’re like, what do you know? Do you have crystal ball? Can you magically see what’s happening out there? It’s like, no. All right, well, Brendan, thank you so much for coming on to Secured.
It’s been like an absolute pleasure to have you come and talk about AI and about what you’re doing at Expo. Is there any parting words or resources that you’d like to share with people if they’re looking at getting into penetration testing or to wanting to give XBOW a go?

Brendan Dolan-Gavitt:
Sure. I mean, so I would say that we’ve got a lot of really cool write-ups with different vulnerabilities that we found with Expo and on Bug bounty things and on open source software. And it’s just for basically all of those, we’ve included the full agent trace of how I thought about things and how I found it. And I think that’s very good for getting a sense of, on a very step by step level, how is the AI actually doing all of this? And you can see it’s not magic, right? It’s running the same commands that you can’t, it’s writing chill script, writing Python programs.
It’s coming up with weird ideas and going off on blind alleys, but it still can also do cool things too. And so, I think that’s a nice way to start with some of it, at least from just getting awareness and exposure as far as being able to start building on this stuff yourself. I think it’s never been easy to do that. You can go grab one of the HNSDKs, hook it up to your favorite tools and say, “Hey, I’m going to go hack you shop.” Try not to over claim and say, “Yeah, I hacked you shop and now the Air Force SKYNET is at hand because okay, everyone hacked your shop. But I think it’s nice that it’s actually so accessible.

Cole Cornford:
Yeah, and I think that it’s going to be very fun and interesting to see how the next couple of years goes. But Brendan, thank you so much for coming on to Secured and when I come over to the States, I’ll let you know. We’ll go hang out, have a beer,

Brendan Dolan-Gavitt:
Absolutely. I’ll probably drinking in Brooklyn,

Cole Cornford:
Thanks a lot for listening to this episode of Secured. If you’ve got any feedback at all, feel free to hit us up and let us know. If you’d like to learn more about how Galah Cyber can help keep your business secured, go to galahcyber.com.au.