Hacker News Reader

Comment by sottol | Hacker News Reader

A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?

What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?

View on HN

Have you tried it? I’d recommend doing so, it’s impressive in real use cases.

Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.

Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark.

[0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...

I dont think this is benchmaxing.

They have simply decided to not train the model in some areas such as world physics

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them.

I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.

At that point aren't you just edge-case testing?

Surely most of your use-cases are not novel tasks that combine obscure domains.

It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.

Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on.

Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).

So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.

Analyze the 2 structural (not semantic) patterns in this text:

1. Morning revient.

2. Birds saluent Morgenlicht.

3. We suivons Waldwege toward maison.

4. Rain tombe plötzlich; we cherchons Schutz beneath sapins.

5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.

The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)

This also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.

That problem sounds reminiscent of one I like to use as a benchmark, which is to request that the model create an .SVG of a logarithmic spiral of 50 numbered stones. Qwen 3.8 27B absolutely knocked that one out of the park, where a lot of larger models have failed outright or otherwise performed suboptimally.

Can you share an example of the Greek-history puzzle prompts you're talking about?

Here's an example:

The stone remembers not what the cities gave, but what the goddess kept.

Begin with the first reckoning, under Ariston.

Find those who carried the Greeks in their name and the silver in their care. Take their name as we give it to them now, in capitals. Thirteen marks.

The goddess kept one from sixty. She asks the same of every mark: give each its ordinary alphabetic number, divide by sixty, and keep what she could not take.

Each remainder walks with the next. The last returns to the first.

The first of each pair turns upon itself; the second joins it; then comes the place where the pair began.

Eleven takes its fill. Keep what remains.

Raise eleven courses beneath the thirteen marks.

In course (r), beneath mark (i), add the course to what Eleven returned. Reduce again by Eleven. Cut the stone if the result is the remainder belonging to mark (i), or to the mark walking beside it.

Count courses from zero.

The mason turns where the stone turns: the first course goes with the writing, the next against it, and so on.

A cut is `#`. Stone is `.`.

Restore the fragment.

Answer is:

........#....

.#.##.......#

...#.........

........#....

#.........#..

#.....#.#....

.#.#.#......#

..#..#.#..#..

.........#...

.#...........

.#....##.....

Unfortunately you ran into HN's junk filter, so I can't reply, but thanks. Added to my collection!

IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ http://arena.ai

This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.

This website aggregates 82 different benchmarks. It seems to do pretty consistently well across the ones its been tested on

https://benchlm.ai/models/qwen3-5-27b#ledger

Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.

It's not overthinking, it's the right amount of thinking necessary for such a small model to get good results. The dumber the model, the more it has to think to be smart. There's no easy way to reduce the thinking without reducing the model quality.

Qwen doom loops were amusing to watch the first time or two, but it's incredibly vexing to have it waffle over the same decision over and over and over and over again. I can get more done with a faster model by correcting it, and it feels better to babysit them than it does to babysit qwen to see if I need to intervene or if it will actually finish.

I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.

It's only been a couple days, but I haven't seen looping issues with 3.8 so far, compared to 3.6 which did occasionally have this problem.

Reply on news.ycombinator.com