Hacker News Reader

Comment by dist-epoch | Hacker News Reader

All of these discussions of models being "nerfed" reminds me of discussions among audiophiles "this cable sounds so much better than this other one, it's night and day, ferrari versus honda civic"

Yet when you do blind tests they can't tell the difference between a $1000 cable and a $1 one.

I bet if you do blind tests between GPT-5.3, 5.4 and 5.5 most would struggle to tell them apart, yet they are certain that "5.5 was nerfed 1 week after release, it's so obvious, it was John Carmack, now it can barely write a for loop"

View on HN

Actually, ELO rankings done blinded on models do vary: https://the-frontier.app, that said, your point looks accurate as far as 5.3 - 5.5 on this chart, 40 to 50 point ELO gain.

I find I have to argue with 5.5 less than 5.3, and I therefore use it when I could reach for 5.3, but I don't think it's a major difference.

Electric Light Orchestra really stole Arpad Elo's thunder.

Exactly this. And it's not really possible to do repeatable trials, it's all just vibes. People have very little awareness of their own cognitive biases.

And companies have high awareness of this all.

They have a way to decrease cost and probably increase token consumption, with gradual changes and no abrupt jump in capabilities, and users have no way to reliably detect it.

Market will advantage companies that do it.

And they are in the best position to automate online narrative shift (the real LLM killer application IMO) towards "Users are imagining it".

You will be amused to hear that when Anthropic "refreshed" 4.6 on AWS Bedrock I found it in my tests and wrote about it – and they actually rolled it back. This is how much non–coding tests may tell you about the model.

So Bedrock 4.6 is old school Opus?

I know you can point Claude code at Bedrock.. might be worth a play.

That's a pretty shallow dismissal, and I bet you $100 I can tell you which model I'm talking to between 4.6 and 4.8 without looking or asking after a handful of messages.

Anthropic famously had a terrible outage back when 4.6 was the latest and greatest, and it was never the same after it came back.

All evidence suggests they simply don't have the compute to keep serving their best models at their most powerful.

I had a repeatable process set up with o4-mini that gave very good results. It generated output in a structured format, and the results were fairly formulaic but good. I did a lot of runs of this process and observed a lot of output.

I set that process down for like 4 or 5 months and then came back to it after GPT 5 came out. o4-mini was literally unusable. I've seen it with my own eyes.

Reply on news.ycombinator.com