Hacker News Reader

Comment by seizethecheese | Hacker News Reader

Instead of the model changing, it’s equally likely that this is a cognitive illusion. A new model is initially mind-blowing and enjoys a halo effect. Over time, this fades and we become frustrated with the limitations that were there all along.

View on HN

Check out this post from a round table dialogue with Greg Brockman from OpenAI. The GPT models that were in existence / in use in early 2023 were not the performance-degraded quantized versions that are in production now: https://www.reddit.com/r/mlscaling/comments/146rgq2/chatgpt_...

Oh interesting. I thought that’s what turbo was.

It was, that's what the comment says?

No it's definitely changed a lot. The speedups have been massive (GPT 4 runs faster now than 3.5-turbo did at launch) and they can't be explained with just them rolling out H100s since that's just a 2x inference boost. Some unknown in-house optimization method aside, they've probably quantized the models down to a few bits of precision which increases perplexity quite a bit. They've also continued to RHLF tune to make them more in-line with their guidelines and that process has been shown to decrease overall performance before GPT 4 even launched.

No. Just to add to the many examples it was good at scandinavian languages in the beginning but now it's bad.

But given the rumored architecture (MoE) it would make complete sense for them to dynamically scale down the number of models used in the mixture during periods of peak load.

It's both. OpenAI is obviously tuning the model for both computational resource constraints as well as "alignment". It's not an either-or.

It definitely got nerfed.

I've never seen "nerf" used colloquially and today i've seen it at least a half-dozen times across various sites. Y'all APIs?

it's popular with gamers to describe the way certain weapons/items get modified by the game developer to perform worse.

buffing is the opposite, when an item gets better.

I've heard nerf used colloquially since like the 90's.

Different circles. I imagine they don't game.

Yep. It's amazing how people are taking "the reddit hivemind thinks ChatGPT was gimped" as some kind of objective fact.

Reply on news.ycombinator.com