Hacker News Reader

You're too kind. Even the CEO of Google retweeted how well Gemini 2.5 did on Pokemon. There is a high chance that now it's explicitly part of the training regime. We kind of need a different kind of game to know how well it generalizes.

I have a draft doing this with text adventures: https://entropicthoughts.com/updated-llm-benchmark

There were no such writeups, 99% of the discussion about difficulties in Crystal were in twitch and discord chats where Google doesn't scrape. (It hadn't yet gotten the public attention that Claude and Gemini's runs of Pokemon Red and Blue have gotten.)

That said, this writeup itself will probably be scraped and influence Gemini 4.

Isn't that the point of a new model anyway?

Yes. Sort of.

Just don’t confuse it with a random benchmark!