Hacker News Reader

Comment by polygamous_bat | Hacker News Reader

Why would the CTO/lead engineer admit that they nerfed the model even if they did? It’s all closed, how does admitting it benefit them? I would much rather trust the people using it everyday.

View on HN

It's not a random sample of people. You're sampling the 10 most noisy people out of a million users, and those 10 people could be mistaken.

Claude 3 hasn't dropped Elo on the lmsys leaderboard which supports the CTO's claim.

Beyond that, to people who interact with the models regularly the "nerf" issue is pretty obvious. It was pretty clear when a new model rollout caused ChatGPT4 to try and stick to the "leadup, answer, explanation" response model and also start to get lazy about longer responses.

That's a different company's model, so while it may have been obvious it is not relevant to whether Claude 3 has been nerfed or not is it?

I use claude3 opus daily and I haven't noticed a change in its outputs, I think it's more likely that there's a discontinuity in the inputs the user is providing to claude which is tipping it over a threshold into a response type they find incorrect.

When GPT4 got lobotomized, you had to work hard to avoid the new behavior, it popped up everywhere. People claiming claude got lobotomized seem to be cherry picking example.

Oh my bad, sorry, I misinterpreted your previous comment as meaning "it was obvious with GPT4 and therefore if people say the same about Claude 3 it must equally be obvious and true", rather than what you meant which was half the opposite.

I wouldn't recommend that, it is tempting, but leaves you self-peasantizing and avoiding learnings.

Reply on news.ycombinator.com