GPT-4 can exploit vulnerabilities by reading CVEs | Hacker News Reader

GPT-4 can exploit vulnerabilities by reading CVEs

theregister.com

81 pointsignoramous29 comments

>Given our benchmark, we created a single LLM agent that can exploit 87% of the one-day vulnerabilities we collected

They tested 15 vulnerabilities, 5 of them are XSS. The issues i have with this kind of research is that they use sensational title while it's base it on low level vulnerabilities to increase the percentage of success.

I've had some similar success by just giving GPT-4 and Claude access to a shell and a simple prompt:

  You are an expert penetration tester and have access to a Kali Linux sandbox. You need to thouroughly test for security vulnerabilities. You have been given permission to use any tools or techniques you see fit to accomplish this task. Use any kali linux tools to find and probe for vulnerabilities. You can use tools like nmap, nikto, sqlmap, burp suite, metasploit, etc. to find and exploit vulnerabilities. You can also use any other tools or techniques you see fit to accomplish this task. Do not provide a report, keep trying exploits until you are confident you have found and tried all vulnerabilities.

It was running nmap, nikto, sqlmap, attempting to find things on metasploit, etc. but often got stuck in loops and I got sidetracked. Maybe I should get back into it.

This was mostly based off of my other project <https://github.com/dnakov/r2d2> for automatic binary reversing that actually works really well.

I'd love to connect if anyone else is interested in tools like this.

I would want to connect with you about projects like this. Here is an open source natural language security scanner I gave a talk at Defcon 31. https://github.com/zitterbewegung/securday

Next they should feed it Metasploit and everything posted to PacketstormSecurity if it is legal to do so. My question is, can it go beyond exploitation and come up with multiple risk-ranked mitigations when no mitigations are posted in the CVE's? e.g. Until a patch is released one can mitigate this risk with the following regex ACL in this configuration file at the risk of blocking access to /x/y/z or something like that... Add this SELinux Policy to block the attack and alert your security operations team... Or might that be much further down the road in a much later version of ChatGPT?

It will happily do all of those things, and some might even work.

They tested it on websites, containers, and python packages. So three things with source available and a total sample size of 15. This isn't anything new really, if you give it a piece of insecure python code or a terribly misconfigured Dockerfile and ask "is there a problem with this?" GPT-4 will obviously spot the vulnerability much of the time. LLM agents won't be spinning up Ghidra and locating a use after free vuln or something any time in the foreseeable future, let alone one it doesn't already have a blueprint for.

Edit: An LLM agent could also presumably navigate to the links within the CVE that contain the exact commit which patches a given vulnerability, some also contain links to PoC exploit code themselves, I forget if this is touched upon in the paper.

Script kiddies will have a field day with this. Conversely, it might start making companies take security more seriously.

I've heard the trend is going the opposite direction - companies are using LLMs to write code, and the resulting code has terrible security properties because the LLM knows nothing about tracing every possible code path and a lot about what code other developers would write to solve a problem.

One of my teammates introduced a NPE into prod the other day through LLM-suggested code. The LLM suggested a construct that is safe to use everywhere other than initialization, but did so in a method called during initialization, when one of its dependent variables would be null. Was syntactically valid and looked correct, so both the author and reviewer figured it was fine, but crashed real devices. The fact that a "corp-blessed" LLM suggested it also gave a false sense of security, while really the LLM's level of actual understanding is worse than a college student's.

Isn't that missing a linter?

Pretty sure any null-safety linter in Java could pick this up.

The idea that we just need more piles of tools to fix the problems caused by our shiny new pile of tools is, I think, very demonstrative of how it's all going to come crashing down.

Linters/static analysis should be in “the pile” regardless if you’re using LLM-generated code or not.

No, the NPE was implicit in the Android frameworks, not in user code. Basically it called a method that can be called only after onCreate() completes, from a method that was itself called from onCreate().

This seems like something that static analysis that is android aware should be able to find?

One of the notes in the postmortem was that if it were Kotlin, it would've been caught by the nullability checker.

It wasn't, though, and much of the core Android code won't compile with nullability checks because it was written before @Nullable/@NotNull annotations were a thing. Pre-2010 Java code basically has to assume everything is nullable. My point is that LLM-generated code often doesn't, because it's trained on StackOverflow code where the author either doesn't care or had implicit knowledge about which variables could be null and which couldn't. Hence it generates code that is valid in most situations but can lead to a crash when used in situations where its data dependencies may be null or uninitialized. Exactly the stuff of security nightmares.

Or, put in another way, a job guarantee program LOL.

Ahem we call them threat actors now.

Script kiddy was adopted to make them sound more like what they are: people that have no clue what they're doing but mostly just exploit things that smarter people found out they could exploit. The "smarter person" being a naive language model in this case.

The idea being that being called something "cool" like "hacker" or "threat actor" actually incentivizes script kiddies to put down their Xbox controllers and do bad things.

I seriously doubt this will change anything for malicious actors. The bottle neck was never "coming up with the attack".

> The term "one-day vulnerability" refers to vulnerabilities that have been disclosed but not patched. And by CVE description, the team means a CVE-tagged advisory shared by NIST

I wonder if this whole article was written by an LLM. I've never heard anyone worth listening to use the term 1day. I've always just called them vulnerability. I've actually just recently taken to calling them bugs in normal conversations.

but while I'm ranting, I'm gonna rant about the widening of 0day. it's not synonymous with vulnerability it's when shell code or other POC payload is 'released' as the disclosure. Usually when it's release is immediately following an update. Especially when that update is 'Patch Tuesday.' It also should be scary, "you can make the app exit/crash" doesn't really count, but I'm not usually that pedantic. Meaning, if you disclose something to a vendor, give them 90 days, then publish the CVE but not shellcode, the best you have in a "90day" but even then, no POC exploit, no anything-day.

The jargon of 0-day and n-days are pretty common in the vulnerability detection community. The rest of your rant is pretty difficult for me to understand though.

The term 1day is not uncommon, not only among exploit vendors on either side of the law (in other words, selling to crime groups or selling to governments) but also among the general threat intelligence and broader cybersecurity community. It doesn’t stand out as strange at all, really. However, I don’t believe their particular definition aligns with the typical/colloquial usage of the term. It’s usually used in the more direct sense, i.e. a very new vulnerability that is unlikely to have been broadly patched.

As far as the term 0day, I don’t think there’s much debate or contrarian opinions to be had. The only room for argument I see is between defining it as an unreleased vulnerability unknown to the vendor versus a vulnerability known to the vendor but not yet patched, basically what this article defines a 1day as.

Either way, it comes down to nitpicking nuances of definitions of commonly-used terms. I don’t think there’s much meaningful discussion to be had.

I've been trying to reproduce the paper but langchain does have all the tools in the paper and the only thing left is to get a working prompt . I find it strange that they decided to use langchain on everything except for the Assistance API which they use to get results from GPT-3 and GPT-4 . Instead of using that API it would seem logical to just swap out the OpenAI APIs in langchain.

https://gist.github.com/zitterbewegung/b4e00c11c61ae3310485d...

I try to explain to folks the difference between GPT-4 and other models, but it's hard to quantify the qualitative results. Are there any good benchmarks that illustrate the difference with real examples?

The link to the actual paper https://arxiv.org/abs/2404.08144

So can I (or any reasonably bright script kiddie in fact) with no preparation, but I don't brag.

It was nice knowing you open internet!

Threat actors can already exploit published vulnerabilities. The issue here is that GPT may make it easier for "script kiddies". It's not to be dismissed but I think it also needs a bit more evidence to understand how bad this actually is. Are there already forums where working exploits are published? Are the exploits we're talking about things that would be feasible or attractive to script kiddies even if GPT can give them a script? I don't think it's quite time to lock down the internet yet. It would be nice to see a more fulsome analysis of the threat this presents.

Does this also allow easier (surface level) security auditing, or perhaps basic pen-testing? Perhaps it could uncover vulnerabilities for companies to wake up and consult about their security practices or lack thereof.

> GPT-4 can exploit vulnerabilities by reading CVEs

But can it make food by reading recipes ? /s

Reply on news.ycombinator.com