Hacker News Reader

Hacker News

Comment by goldenarm | Hacker News Reader

Parent Full thread

goldenarm

Geometric mean of all these benchmarks :

* GPT-5.6 Sol: 65.5

* Fable 5 (w/ fallback): 64.5

* Opus 5: 64.0

* DS-V4-Pro 0813: 62.5

* Kimi-K3: 62.3

* DS-V4-Flash 0731: 55.8

* GLM-5.2: 47.3

View on HN

Maybe it's me but I don't see how DS Flash is better than GLM at all, much less by a huge gap. I'd probably protest less against Fable and Opus being put at the same level than many would, but there's no denying the two models are a very different experience from each other. I guess where I'm going is no one should pick a model by the benchmarks.

I think instruction following carries outsized weight in these evaluations.

I'm not the most LLM-savvy person around, and I'm not gonna say I've put a ton of effort into practically compared these open models. But, a month or two ago I did do some "practical evaluates" testing GLM 5.2 versus DSv4 (flash/pro) with OpenCode's subscription with some late 80s Unix clone-type work, and this jives with my experience.

GLM ended up being far slower, and far more expensive, for approximately the same results. There was never a problem that GLM could solve that DS couldn't solve, faster, and significantly cheaper.

I strongly agree that you shouldn't pick a model based on benchmarks. But for me, I found GLM really underwhelming given its cost and speed.

DSv4 isn't as good as GPT or Claude or what have you, but it's fast, and pretty darned effective. I can run a 3-bit quant of DSv4 locally on my system with ~15 tokens per second, and for a local model it might be the most overall effective at coding. For what it is, it's extremely impressive.

My experience is the same.

Imo it has a lot to do with you/the harness tries to get it to test itself. Deepseek v4 flash seems more than capable of understanding when something has failed, and making changes until it works. I've definitely seen it make mistakes I would expect something like Opus to find, but it works through them on it's own (and for literal pennies).

At the end of the day, I think that's one of the most important features of a model.

Exactly the same experience. I really loved GLM5.2 for a while, but after trying it again after riding DSv4 flash (new) for a while, they're mostly at the same capabilities, with GLM being slower and much, much more expensive. A task cost me 2$ where it did very wrong, whereas Flash nailed it almost instantly for like a rounding error on my billing page.

Interesting!

GLM 5.2 is slower for sure (although they offer a fast version), and it's more expensive. But in my experience, it's universally better than Deepseek V4-flash-0731. Don't get me wrong, the new Flash version is amazing.

But the use cases I have looked at are about source code understanding, bug finding, etc. - GLM 5.2 is clearly better.

I think by using some prompt engineering, you will probably be able to close this gap, but some extra work is needed.

And I'll say it again: the new Flash version is amazing. I love it. That level of intelligence for the price is unprecedented, and the fact that it's open weights and runs locally makes me genuinely happy.

In my little social circle DS4F generally substitutes for GLM 5.2 except it's the next best thing to free.

It isn't. I run both at home. GLM5.2 Q4 crushes DSv4Flash0731 Q8. I reach for DS for speed and for medium effort level work. If I care about quality I'll reach for GLM5.2 Looking at this release, I'm comparing it to GLM5.2 and it seems to beat GLM5.2, only time/experience will show. If true, then I'm happy. It's much easier to run than Qwen3.8/KimiK3

Reply on news.ycombinator.com