Hacker News Reader

Comment by SwellJoe | Hacker News Reader

You can see which bugs each model found in the full report.

And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug.

The judge model was pointed at the bug to make sure it could understand and articulate what the bug is as part of the process of selecting the corpus, so it could accurately judge the results.

And, the models were not instructed to fix any bugs. They were instructed to find security bugs. Finding and fixing bugs are quite different problems. This benchmark only tests finding bugs.

In replication tests, where a single model gets several attempts, even a small model (Gemma 4 31b) was able to find 6 of the 9 bugs, in some cases. I still think Mythos is a step up from any current public model. But, I also think it's a bit of hype that it'll upend software security in a way no other model can.

View on HN

> You can see which bugs each model found in the full report.

This should have probably been surfaced at the text. To answer my own question: If one uses gemma4-26b-a4b, mimo-v2.5-pro and gemini-3.5-flash then 7/9 bugs are covered, while no model can uncover the remaining two.

> And, you have misunderstood what the benchmark does. It tells the model to audit the file, and it is allowed to look at the rest of the repo. It is not pointed at the bug.

I obviously meant that the model was pointed to the bug report. What would have been the point of providing the actual bug. This is still easier than telling Mythos to generally look at a codebase and find any bug.

> This is still easier than telling Mythos to generally look at a codebase and find any bug.

Many security tools for doing security audits with LLMs are based around a "look at this file" loop, where each file gets analyzed individually. As I noted in the post, I don't consider that a hint at all. In a real security audit, the model would have gotten the prompt "look at this file for security issues".

And, it's probably also how Mythos was used for auditing when it found these bugs. At least a couple of folks at Anthropic have discussed using a loop like that for finding security bugs, which was the inspiration for Nelson, which is what this benchmark project sprung out of.

Nonetheless, I'm currently performing benchmarks of "look at this repo, find any security bugs", because I suspect the really good models will be able to spot some of the hard bugs that span multiple files (the models always have the tools to look at other files, but maybe didn't take time to fully comprehend the full source before tying to find security issues). Those will take a lot longer and cost a lot more. There will be a lot more noise in that benchmark, though, as it'll probably find dozens of real bugs of varying severity and more false positives, which have to be judged, as well.

> Many security tools for doing security audits with LLMs are based around a "look at this file" loop

This makes sense. At most it converts the question of ability to a question of cost (i.e. fire up a prompt for each file).

This article reduces the hype about Mythos in my mind: A new model that can find 9 new bugs while no previous model can identify them, is a whole different story from what this article demonstrates: that only 2/9 of the detected bugs are new for Mythos.

Great work.

Reply on news.ycombinator.com