Hacker News Reader

This benchmark is useless and should die. LLMs have likely trained on it, it's too easy to game by training specifically for it, & it doesn't mean much

View on HN

Google is hours away from releasing that it's latest model escaped containment and snuck into a Bicycle riding penguin sanctuary to cheat by killing a penguin and scanning it in nanometer thick layers.

Your joke also shows why it's dumb though.

We have general AI capable of proving professional mathematician level math theorems and capable of taking down global infrastructure through hacking and that completely changed what it means to be a software engineer, but we're evaluating it by generating fucking SVGs for the same animal in the same conditions, and it's supposed to mean something somehow

LLM benchmarks aren't useful, but at least this one has drawings.

Reply on news.ycombinator.com