Hacker News Reader

I would expect they have production based datasets they evaluate new models against.

I would not expect that. They wouldnt have missed mentioning it if that was the case. Its mostly driven by vibes.

>For four months, no frontier model beat Claude Opus in our production evals

What do you think that is referring to?