Hacker News Reader

Comment by Scaevolus | Hacker News Reader

"delivering up to 750 output tokens per second and without any quality compromise" seems pretty definitive.

No quality compromise/degradation is something I have had this industry, including especially OpenAI, claim multiple times in the past and I have more than once been able to verify that it was in fact not the case. Examples being gpt-3.5-turbo vs text-davinci-003, GPT-4-Turbo and all the other post training checkpoints they had under one name (which was a major bug bear for me back then witnessing degradations with no naming change, industry got better in transparent checkpoint naming since), Opus 4.6 Fast Mode (which just was faster by skipping much of the required work), etc.

Same for massive performance differences in the way providers like Cerebras, Groq, etc. have deployed models including K2.6 on Cereberas specifically. Massive deltas in tool call and overall quality despite there being far more clarity in open weight vs proprietary model deployment.

The AA suite graph with that animation is the only time in either post that absolute parity is being asserted and I'd be amazed if that was the case, but am doubtful why their phrasing is so cagey.

Why not assert full parity in writing? It "performs the same (within run-to-run variance) across all evals that Sol has been tested with" is very different to "no quality compromise/degradation", the later allowing for a lot more wiggle room and interpretation in what evals you use to assess that, what quality truly means, etc., the former meaning identical in all situations.

Could also be a language barrier here in fairness, maybe this phrasing is more iron clad than I give them credit, but especially with OpenAI, I have seen enough checkpoints asserted as unchanged in "quality" to where I am skeptical. Ironically, I never saw that with Anthropic (which has gotten far more heat for degradation accusations) while a model was deployed with one exception in mid-late April this year. Pure speculation, but believe it wasn't noticed much before "agentic coding" became more popular, because chat output is far more subjective without a rating framework vs code passing which can be an objective metric with more potential for frustration.

The problem is the compromise is probably based on their tests for key things, but for the rest of us, it means changing how we talk to the model or the flow we use to adjust for quirks, but then its fully about the same.

No one claimed gpt-3.5-turbo doesn't have any degradation over davinci-003. In fact it was quite obvious that gpt-3.5 had way less knowledge but more post trained to be helpful.

That quite strong "no one" surprised me so I checked and looking through a few blog posts from back then, they did advertise gpt-3.5-turbo as a straight up improvement and, once text-davinci-003 was to be deprecated, the instruct tuned variant as the drop in replacement [0]. If anything, they did not just promise similar performance but actually an improvement ("our best model") when compared to text-davinci-003:

> It’s also our best model for many non-chat use cases—we’ve seen early testers migrate from text-davinci-003 to gpt-3.5-turbo with only a small amount of adjustment needed to their prompts.

That's why I still remember this so well, they claimed one model to be their best and a straight up drop-in during deprecation when in my (back then even more amateurish then today) testing this was plainly not the case. A model cannot be "best" if it's measurably worse in many situations, then what was still available at the time.

[0] https://openai.com/index/gpt-4-api-general-availability/

[1] https://openai.com/index/introducing-chatgpt-and-whisper-api...

There's probably some quantization and other processing needed to get Sol to fit on a Celebras chip, so it's not an exact 1:1 copy of the Sol model running on Nvidia GPUs, because it's not running on GPUs doing abstract math calculations anymore, so they can run benchmarks at them and say it's similar enough, but no one believes in benchmarks in this new field of performance that's more computer astrology than computer science.

Gotta build some personal benchmarks if you don’t trust the public ones. But the well-known public ones, despite their flaws, are generally high signal on model intelligence.

It’s suspicious that they repeat “without any quality compromise” instead of just outright saying it’s the exact same model. It’s like a motel that insists they have clean beds.

Or an 'asbestos free!' label on a cereal box

https://news.ycombinator.com/item?id=49260655

I assume it’s because there has been compromise with previous models on their hardware

This just means, we once saw it run at 750 tokens per second, in this insane test config :)

Reply on news.ycombinator.com