Hacker News Reader

I read recently that small variations in the tests cause failures by large margins.

If this doesn’t show over fitting in don’t know what would.

Eventually, all the better AGI tests should have large private evaluation datasets with no possible cheating or feedback loops. We're getting there.

Wasn’t that for human tests, i.e. not specifically AI benchmarks? Benchmarks should generally not be game-able by overfitting.

The article shows all the tests against human performance.

The math one in particular is the one where small variations reduce the success rate significantly. I can’t find the source but it was pasted here in the last 2 weeks.

You're probably remembering this: https://news.ycombinator.com/item?id=42565606

Reply on news.ycombinator.com