Hacker News Reader

Comment by mh- | Hacker News Reader

Not nitpicking, but they said:

> we never intentionally degrade model quality as a result of demand or other factors

Fully giving them the benefit of the doubt, I still think that still allows for a scenario like "we may [switch to quantized models|tune parameters], but our internal testing showed that these interventions didn't materially affect end user experience".

I hate to parse their words in this way, because I don't know how they could have phrased it that closed the door on this concern, but all the anecdata (personal and otherwise) suggests something is happening.

View on HN

"Anecdata" is notoriously unreliable when it comes to estimating AI performance over time.

Sure, people complain about Anthropic's AI models getting worse over time. As well as OpenAI's models getting worse over time. But guess what? If you serve them open weights models, they also complain about models getting worse over time. Same exact checkpoint, same exact settings, same exact hardware.

Relative LMArena metrics, however, are fairly consistent across time.

The takeaway is that users are not reliable LLM evaluators.

My hypothesis is that users have a "learning curve", and get better at spotting LLM mistakes over time - both overall and for a specific model checkpoint. Resulting in increasingly critical evaluations over time.

You’re not wrong, but, I can literally see it get worse throughout the day sometimes, especially recently. Coinciding with Pacific Time Zone business hours.

Quantization could be done, not to deliberately make the model worse, but to increase reliability! Like Apple throttling devices - they were just trying to save your battery! After all there are regular outages, and some pretty major ones a handful of weeks back taking eg Opus offline for an entire afternoon.

And yet, people's complaints about Claude Code over the past month and a bit are now justified by Anthropic stating that those complaints caused them to investigate and fix a bunch of issues (while investigating potential more issues with opus).

> But guess what? If you serve them open weights models, they also complain about models getting worse over time.

Isn't this also anecdotal, or is there data informing this statement?

I think you could be partially right, but I also don't think dismissing criticism as just being a change in perspective is correct either. At least some complaints are from power users who can usually tell when something is getting objectively worse (as was the case for some of us Claude Code users recently). I'm not saying we can't fool ourselves too, but I don't think that's the most likely assumption to make.

Selection bias + perceptual adaptation is my experience. Selection bias happens when we play the probabilities of using an LLM and we only focus on the things it does really well, because it can be really amazing. When you use a model a lot you increasingly see when they don't work well your perception changes to focus on what doesn't work vs. the what does.

Living evals can solve for the quantitative issues with infra and model updates, but not sure how to deal with perceptual adaptation.

And survivor bias.

People who like the tool at first use it until they stop liking it -> "it got worse"

People who dislike the tool at first do not use it -> "it was bad"

"or other factors" is pretty catch-all in my opinion.

> I don't know how they could have phrased it that closed the door on this concern

Agreed. A full legal document would probably be the only way to convince everyone.

Wording definitely could be clearer.

Intentionally might mean manually, or maybe the system does it on it's own when it thinks it's best.

Reply on news.ycombinator.com