Comment by SyneRyder | Hacker News Reader

Comment by SyneRyder | Hacker News Reader

Parent Full thread

While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:

https://x.com/PawelHuryn/status/2095982259761475945

https://bughunt.productcompass.pm/?preset=all

Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.

(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)

Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.

Still, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.

If model can not automatically reproduce bug, while human manually can... you got a problem in CI.

> Luna Max performing better than Fable 5.1 High

Perhaps you are reading too many benchmarks.

Edit for answer :

I agree Luna is great cheap model.

But if Fable was hitting security limits, yet was still included in benchmarks...

What flies better? Elephant or paper plane. You can make objective benchmark about that. But not much value for logistics company

> Perhaps you are reading too many benchmarks.

Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.

I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.

I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.

Reply on news.ycombinator.com