Hacker News Reader

Comment by no-name-here | Hacker News Reader

In pretty much every single HN post on this topic, there are a number of commenters claiming it’s false. Continued quantifiable data like this seems very important at hopefully resolving the ongoing disagreement about the facts.

View on HN

My read of the zeitgeist on HN is that these new LLMs bring with them a torrent of false or useless security reports, that whatever may be true simply drowns.

The end result is both that there are more critical CVE and that there aren’t.

Yes, there’s been a very popular narrative that Mythos’ abilities are just marketing fluff. I think it’s clear that there’s a real capability here, even if Anthropic’s communications have been heavily influenced by PR concerns.

In pretty much every single HN post ~~on this topic~~, there are a number of commenters that are probably bots. Anyways it makes sense from a theoretical standpoint that LLMs should be able to find flaws in code better/faster than humans eventually, and its reasonable to think that time has come

AI has driven many people into denial. It's excruciating to watch otherwise smart individuals embrace terrible thinking, over and over and over.

I've seen plenty of people saying "Mythos isn't all that exceptional, lots of LLMs can find security vulnerabilities" -- and indeed there is some evidence for that; it sounds like Anthropic was taken somewhat by surprise at how easily a simple prompt managed to get Mythos to deliver exploits and didn't distinguish immediately between the effectiveness of Mythos and the effectiveness of the prompt.

But the claim of "LLMs aren't making a difference in vulnerability discovery" has been laughable to anyone who has been reading security advisories for the past 3 months. Just look at the Credits lines.

I thought the point was not that Mythos finds more vulnerabilities, but that it can exploit them much more successfully. I thought the report showed it didn’t find much more than Opus 4.8. (Or did I misread?)

If you look at public benchmarks like ExploitBench [1], then you'll see this is mostly a question of token budget. Once you give it sufficient tokens to burn, GPT 5.5 is roughly as good as Mythos when it comes to finding bugs and building exploits. With some clever auto-prompting to clear stalls, it even beats the base Mythos version. So Mythos' "magic" is not in the model, but in the harness and compute env. That's probably also why they never released it, because Anthropic already struggled heavily to make Opus available to the general public. Releasing Mythos publicly may well be technically impossible for them due to compute constraints.

[1] https://exploitbench.ai

I still have to see a single glibc bug that truly matters. I don't have illusions about our code quality, so there must be something to find.

We got many high-quality bug reports, some of them with a security aspect to them. Several of them received CVSSv3.1 scores of around 9.8 from the rating agencies, but these high numbers are misleading. (Vulnerability scoring is hard, and it's pretty much impossible for a library without reference to an application that uses the library.) Looking beyond the numbers, everything reported this year (and late in last year) was pretty harmless so far.

Does this mean LLMs are making a difference? For upstream developers, definitely. For end users? Not that much yet.

Maybe the picture changes once the organizations sitting on the good findings figure out how to disclose them to the relevant upstream projects. When I read the announcement of Project Glasswing, I immediately thought that this was going to be the hardest part.

Reply on news.ycombinator.com