> And even though it's super straight forward to collect the evidence, there seems to be zero substantiating the vibe bro conspiracy theories.

Until shit like this: https://www.anthropic.com/engineering/april-23-postmortem

Where people pointed out issues early and en masse, and Anthropic denied it was happening, gaslighted anyone claiming this was an issue, then begrudgingly admitted it was an issue, and then spent another two weeks "fixing it".

Or shit like this: https://www.anthropic.com/engineering/a-postmortem-of-three-...

Anthropic is in a perpetual state of "oops, these 'bugs' degraded our model quality" and only admit the issues when it's immediately obvious and visibly affects a large number of customers.

Otherwise all open benchmarks can be (and are) gamed. And it's quite hard to judge the output of a non-determenistic black box that Anthropic (or OpenAI) constantly tweak.