Professional Vulnerability Researcher here... You are correct. Over the years this industry has seen an influx of script kiddies who do nothing but run tools. It's sad but I really think this field needs more gate keeping...
Some of these AI slop report exchanges are absolutely hilarious. Love seeing people caught red handed then trying to play it off.. This is why Vulnerability Research needs more gatekeeping.
This shouldn't be a surprise. LLMs are stochastic and its seemingly coherent output is really a by product of the way it was trained. At the end of the day, it is a neural network with beefed up embeddings... That is all. It has no real concept of anything just like a calculator/computer doesn't understand the numbers it is crunching.
Unless the blog was Phrack, you shouldn't be surprised I never heard of your blog unless it is something you think I would know.. This stuff didn't have such a public community back then like it does now..
Bro in 2006 there wasn't any blogosphere. You had IRC.. I can’t find anything of the sort and do you have a link to this discovery that you say was made via LLM?
I don’t recall anyone saying anything of the sort back then about fuzzing? Back then you could run the most basic fuzzer and find tons of bugs! Where did you see people complaining about fuzzers??
I might be. Deepsleep also sort of found a bug, but you need to ask yourself… is it doing it better than tools we already have? Could a fuzzer have found that bug in less time? How far along did it really need to be pushed and also.. I have no doubts it probably trained on certain types of bugs for certain specific code bases.. Did they test its ability to find the same bug after applying a couple transforms that trip up the LLM? Can you link me to this article about o3? I have my doubts. I’d love to see the working exploit…
Also if you throw these models at enough code bases, they will probably get lucky a couple times.. So far every claim I have seen didn’t stand up to rigorous scrutiny. People find one bug then inflate their findings and write articles that would make you think they are far more affective than reality and I am tired of this hype.
CURL had to stop accepting bounties after it found nearly all of em were just AI generated nonsense…
Also I stated that they indeed provide very large gains in certain areas. Like writing a fuzz harness and reversing binaries. I am not saying they have absolutely no utility I am simply tired of grifters attempting to inflate their findings for clout. Shit has gotten out of control.
Alright folks.. To qualify myself. I am a vulnerability Researcher @ MIT. My day to day research concerns embedded hardware/software security. Some of my current/past endeavors involve AI/ML integration and understanding just how useful it actually is for finding/exploiting vulnerabilities. Just last week my lab hosted a conference that included MIT folks and the outsiders we invite. One talk was on the current state of AI/LLM. To keep things short, this article is sensationalized and overstates the utility of AI/ML on finding actual novel vulnerabilities. As it currently stands, LLMs cannot even reliably find bugs other less sophisticated tools could have found in much less time. Binary Exploitation is a great place for illustrating the wall you’ll hit using LLMs hoping for a 0day. While LLMs can help with things like setting up fuzzers or maybe giving you a place to start manual analysis, their utility kind of stops there. They cannot reliably catch memory corruption bugs that a basic fuzzers or sanitizers could have found within seconds. This makes sense for that class of bugs. LLMs are fuzzy logic and these issues aren’t reliably found with that paradigm. That’s the whole reason we have fuzzers; they find subtle bugs worth triaging. You’ve seen how well LLMs count, it’s no surprise they might miss many of the same things a humans would but fuzzers wouldn’t (think UaF, OOB, etc). All the other tools you see written for script kiddies yield the same amount of false positives they could have gotten with other tools that already exist.. I can go on and on but I am on shuttle, typing on small phone. TLDR: Article is trying to say LLMs are super hackers already and that’s simply false. They definitely have allure for script kiddies. In the future this might change. LLMs time saving aspects are definitely worth checking out for static binary analysis. Binary Ninja with Sidekick saves a lot of time! But again. You still need to double check important things!
The issue is, though ChatGPT and the likes definitely speed up development in one respect, they create a bottleneck in various other places.. “vibe coders” have essentially no idea what the code their LLM cranked out even does and are unaware the code they generated is often riddled with bugs yet still looks plausible. Not only that, they are often trying to develop things, that for the most part you could have just googled. Of course an LlM can write your CRUD app to a pint but then try to add a novel aspect and you see it pretending to reason and make progress when in reality you’re just spinning your wheels. I do research at MIT, my day to day involves developing systems/solutions/technologies that did not previously exist. When the bulk of your work lies in novel things such as R&D the limits of LLMs become very apparent and they offer no further help.
Every single day friends of mine ( many who aren’t programmers just wanna be CEOs of the next trend) tell me about these “project” of theirs. At the beginning it amazed them. They got a GUI up displaying some numbers up in Python with just a few queries! So cool! To them this probably looks like black magic. As the project grows to anything beyond the trivial, they cannot push it further along. They can’t look at the code themselves to fix the minor issue any human would immediately see because they have no clue/haven’t read the bulk of generated code (often trash quality riddled with vulnerabilities)… They often resort to trying a million different tools and platforms, spinning in circles, praying another model can fix their abomination… You get the idea. Sorry for poor grammar typing on small phone on bus.
Was I the only one confused by this? It wasn't just me right? I love when I see things like this. "The cool thing about our kernel is that you cannot share memory! It's super secure. Except for, you know, ..." then list nearly everything. What were they trying to provide/gain with this proposal?
I remember when I was doing iOS/macOS security research back in 2015 and I though to myself "I bet core audio has bugs.." but never really looked into them because my thinking was they wouldn't have been too too useful unless you can get someone to open an audio file... But great work.
Ton of work already being done on this. I am a Vulnerability Researcher @ MIT and I know of a few efforts, just at my lab alone, being worked on. So far nearly everything I have seen seems to do nothing but report false positives. They are missing bugs a fuzzer could have found in minutes. I will be impressed when it finds high severity/exploitable bugs. I think we are a bit too far from that if its achievable though. On the flip side LLMs have been very useful reverse engineering binaries. Binary Ninja w/ Sidekick (their LLM plugin) can recover and name data structures quite well. It saves a ton of time. Also does a decent job providing high level overviews of code...
I agree and binary exploitation/Vulnerability Research is my area of expertise.. The whole "Lets port everything to Rust" is so misguided. Binary exploitation has already gotten 20x harder than say ten years ago.. Even so.. Most big breaches happen because people reuse their password or just give it out... Nation States are pretty much the only parties capable of delivering full kill chains that exploit, say chrome... That is why I moved to the embedded space.. Still so insecure...
Fair point. I would say no. Even getting paid extra for extra work.. I would still be at an energy deficit because our tethering of energy to the amount of money we make is somewhat artificial.. There are limits on how much control we have of ourselves no matter how much money someone gives us. To make things extreme for the sake of clarity.. Whether I am paid a billion dollars a day or one dollar a day, I know I will need at least eight hours of sleep either way (to operate at full capacity)...
I often ponder if I have the energy to go back to school. I am employed by MIT at one of the labs where I do research for embedded security. As a consequence, they offer free classes you can pick up. I am yet to actually take advantage of that yet but your comment has me thinking the same thing. I turn 36 in a couple days!
Neat.. But.. Why? There are already a ton of DAWs and at this point all of them essentially support the same features and folks end up using all the same Waves VSTs... DAWs can be resource intensive when low latency is key (say recording vocals). Browser DAW ain't gonna do so well for that type of thing. Still cool though..
Likewise! I can understand drugs and such… But gambling? At least with drugs you have a guaranteed reward. You purchase heroin and you use it, you’re high. Task accomplished gambling it like “here’s money so I might possibly get high if I’m lucky.”