Hacker News Reader

Comment by Majromax | Hacker News Reader

> [A]ll of them found some that the others hadn't discovered. Now, correctness issues aren't the same as vulnerabilities, but the same principle about using heuristics to find defects applies.

This makes perfect sense, but that conflicts with the impression put forward by Anthropic and OpenAI (in particular) that they alone occupy 'frontier model' spots. Frontier models should large dominate their competitors on a capability basis, but if GLM 5.2 (now 5.3) is routinely finding bugs / vulnerabilities missed by Fable and Sol then GLM might be genuinely a frontier-grade model by itself.

View on HN

“Company hypes own product, downplays competitors” is still a thing with AI

> This makes perfect sense, but that conflicts with the impression put forward by Anthropic and OpenAI (in particular) that they alone occupy 'frontier model' spots.

Not necessarily. Even near the frontier, we don't really have a total ordering of capabilities, but a partial order. And even frontier models make plenty of mistakes. Combined with the randomness inherent in searching large codebases for vulnerabilities or correctness issues, it is entirely plausible that even much weaker models (and GLM-5.2 isn't even weak) can stumble upon issues that stronger models missed.

My current hypothesis – for which I have only limited evidence, unfortunately – is that it is better to have multiple reasonably powerful (but not necessarily frontier) models looking for issues than just one very powerful one. And even then you're likely to miss out on some issues.

Fable 5 is just over two months old.

For normal software it would be as you say, but LLM progress is so ridiculously fast that things go from "bleeding edge" to "eh, you'll do" in about that timeframe, and "eh, you'll do" to "why even bother with this old rubbish?" in the same again.

Or, from a different perspective, we can expect some new frontier model from Anthropic in a week or two, and from OpenAI in a month or so.

There’s no guaranteed progress, tho. Opus 5 is a regression.

Reply on news.ycombinator.com