Hacker News Reader

Comment by shelled | Hacker News Reader

How much do you get out of these models for same amount (say $20 sub for claude/codex/zai-glm) - the equivalent amount of tokens? Also is deepseek as good as glm5.3? (if we don't compare it with frontier models from USA)

View on HN

Well, to be honest it’s all based on my subjective experience. I started benchmarking models on 10 of my real tasks ranging from easy to hard but even with the same model the completion time sometimes varied by as much as 50% between runs. That made me realize you’d probably need hundreds of tasks to get any meaningful numbers. Otherwise there’s just too much variance. So I can’t give you solid benchmarks without spending weeks testing everything properly. What I can say is that I actually use most of these models regularly for different things, so I’ve developed a general feel for them. DS is my main model for work, Claude for hobby projects and local AI, Fable mostly for design, Codex for various other tasks and sometimes Grok for health-related stuff. Cost-wise, a $200 subscription wasn’t enough to get through 20-30 tasks while $30 of DeepSeek was, and that was with Sol, not even Astra. Fable is expensive too and I haven’t found it particularly strong at coding. Opus 5 was horrible in my experience, though maybe 5.5 will be better since I’m testing it now. I also ran GLM 5.3 Flash locally for a while but it was pretty slow and its code was usually worse than both DS4.1 and Qwen Flash Next. Things are moving so quickly that it’s honestly hard to keep up, so take all of this as my general experience from actually using these models rather than a proper benchmark.

> Cost-wise, a $200 subscription wasn’t enough to get through 20-30 tasks while $30 of DeepSeek was

Thank you. This alone helps! I think I should jump into it once and try it all out while I still have GLM access at old prices as a backup.

Np, good luck! PS I’ve been using Sol 6 and Opus 5.5 today and they’ve been great so far. The new Opus is smart, fast and cheap. It makes mistakes but after every task I ask Sol to review it and it catches most of them. I need to test the models properly but it looks promising so far! Hope they won’t nerf them again. I think DS may still be cheaper though (and it’s 300 TPS via the API which is sick).

Reply on news.ycombinator.com