GPT-4 autonomously hacks zero-day security flaws with 53% success rate | Hacker News Reader

GPT-4 autonomously hacks zero-day security flaws with 53% success rate

newatlas.com

13 pointsharambae19 comments

A zero-day exploit is a cyberattack that takes advantage of a previously unknown or unaddressed security flaw in a computer's software,but What's the point of this claims without any evidence?

Article claim things without any evidence or poc given.

To be fair, they provided CVEs. And at least one of these seems worth attention, i.e. is related to a popular product (PrestaShop):

https://nvd.nist.gov/vuln/detail/CVE-2024-34717

So basically LLMs can be used as more sophisticated fuzzers, and that's a good thing - although I'd say this way of testing is still quite expensive.

Those are only web vulnerabilities ("non-web vulnerabilities require complex environments"), and they used specific "agents" that they primed with the specific exploitation tools they'd need, "5-6 documents" about that tool and vulnerabilities, and a specific prompt. As far as I can see, they don't disclose any of those. They also didn't pre-register the study in any way.

Honestly this paper is missing almost all of the write-up.

Looking at the software they exploited...

The first one looks like a student project? https://www.sourcecodester.com/php/17115/travel-journal-usin... I see 79 lines of PHP code. It has two buttons, "read journal" and "write journal", to make a plain-text post. How did this get a CVE assigned?

The next two are from a CMS with a README in Lithuanian, 2 stars, and a dead website.

I don't want to dunk on all those projects, and maybe those kinds of programs are the bulk of CVEs, but they managed to find projects that did absolutely no validation and which can get a XSS/CSRF/SQLi on the first attempt (just put some JS or SQL into any form field and you've won).

Is that really a random selection of 14 CVEs? Also why does the table have 14 when the text claims 15?

In the paper, it seems they really stretch the definition of "zero day" to mean †not part of the GPT-4 training data*. In another words, these were exploits that GPT-4 was not trained on.

Paper is linked at end of article.

https://arxiv.org/abs/2406.01637

These are always so unclear; it seems like this is GPT-4 hacking computers by using zero-days that are already known in some form. What GPT-4 is NOT doing is discovering novel vulnerabilities and then also exploiting them, is that correct?

Zero Day means unknown, but at this case, I call it bullshit until they give us evidence. The article mention Devin, the scam project as an example too lol

That’s how I read it. GPT was given the CVE list and could use zero days accordingly.

Big fan of the word "autonomously" here. Like it's skynet or something

It's being used in the sense of an autonomous munition here.

I understand. But in the context of a headline it feels like a deliberate choice to sensationalize.

John Mitchell, a seasoned veteran in the bustling newsroom of the Daily Ledger, had seen it all. From political scandals to human interest stories, his pen had chronicled the ebb and flow of the world for over three decades. Today, he sat at his cluttered desk, the rhythmic clatter of typewriters around him, drafting a piece about a groundbreaking yet alarming development in the tech world. Scientists had unveiled a hacking autonomous munition AI, a sophisticated machine capable of exploiting 53% of real-world vulnerabilities in zero-day settings. It was a story of immense technical achievement shadowed by its potential for misuse.

Mitchell’s fingers flew over the keyboard, his words painting a balanced picture of innovation and caution. He handed the draft to his editor, a young, ambitious man named Trevor Hayes, who scanned the article with a furrowed brow. “This is good, John,” Trevor said, tapping his pen against the desk, “but it needs a punchier title. Something that grabs the reader's attention. How about ‘Rogue AI Threatens to Unleash Cyber Chaos’?”

Mitchell sighs, opens the bottom right drawer of his desk, and takes out a half-empty bottle of amber-colored liquid.

Can you do another only this time in the style of Brian O'Nolan as Flann O'Brien?

I'd lightly edited the first one, but this one is all ChatGPT with an Original Thought Capability Module installed:

In the dimly lit newsroom, where cigarette smoke intertwined with the hum of typewriters, grizzled old hack John Mitchell wrestled with the beast that is truth. Clad in a rumpled trench coat and armed with a trusty pen, he carved out an exposé on a monstrous contraption birthed in the bowels of some godforsaken lab. The contraption in question? A hacking autonomous munition AI, cunning as the devil and twice as dangerous, capable of worming its way into 53% of real-world vulnerabilities with the finesse of a cat burglar.

His fingers danced across the keys, painting a picture stark and sober, a testament to his years of journalistic grit. But no sooner had the ink dried than his editor, Trevor Hayes, the greenhorn with a nose for sensationalism, barreled in, waving the draft like a flag. "John, this is solid," he quipped, "but it needs more pizzazz. How about ‘Killer AI Set to Unleash Cyber Armageddon!’?" Mitchell’s shoulders sagged under the weight of compromise, for he knew that in the dance between integrity and sensationalism, the latter often led. With a sigh, he glanced at the clock, pondering the fine line between truth and the grotesque mask of hyperbole that news demanded.

Cheer's for the response, both forms are entertaining but I fear I'd mistake neither for work by the author of An Béal Bocht, The Third Policeman, and At Swim-Two-Birds.

It's not just language style that requires capture, or rather a particular style of smashing two languages together in a steam bath, it's also a manner of logic tangential to both the norm and to Spike Milligan.

Whenever I see these “researchers get LLMs to succeed at X complex task” I wonder why I can’t even get it to do something basic in the real world (at my job). The gap seems vast

There is technique to getting what you want out of LLMs.. Of course there is "prompt engineering", Chain of Thought, etc. In general this idea of "unlocking" is referred to as "unhobbling"

I get value from LLMs just about every day, mostly through the Copilot plugin for VSCode.

Yesterday, for example, I had some ts React class-based components I'd written which I wanted to convert to functional components. Select all, "convert these class components to functional". Done. Took a couple minutes to do this to a handful of components. A little manual cleanup after. This would have taken a lot longer manually.

Reply on news.ycombinator.com