Hacker News Reader

Comment by futureshock | Hacker News Reader

I know what you are saying, but I very much disagree. There are also better chess engines. That’s not the point.

It’s all about the “G” in AGI. This is a nice demonstration of how LLMs are a generalizable intelligence. It was not designed to play Pokémon, Pokémon was no special part of its training set, Pokémon was not part of its evaluation criteria. And yet, it plays Pokémon, and rather well!

And to see each iteration of Claude be able to progress further and faster in Pokémon helps demonstrate that each generation of the LLM is getting smarter in general, not just better fitted to standard benchmarks.

The point is to build the universal hammer that can hammer every nail, just as the human mind is the universal hammer.

View on HN

It is not generalizable intelligence, its wisdom of the crowds. Claude does not form long term strategies or create predictions about future states. A simpler GOAP engine could create far more elaborate plans and still run entirely locally on your device (while adapting constantly to changing world states).

And yea you could have Claude use a GOAP tool for planning, but all you’re really doing is layering an LLM on top of a conventional AI as a presentation layer to make the lower AI seem far more intelligent than it is. This is why trying to use LLMs for complex decision making about anything that isn’t text and words is a dead end.

> It is not generalizable intelligence, its wisdom of the crowds.

Did you see twitch chat plays pokemon? There was not much wisdom in that crowd :P

Well, we know that some ways to organise crowds work better than others.

Pokémon guides were definitely part of every LLM training set. Game is so old, there are thousands of guides and videos on the topic.

LLMs will readily offer high quality Pokémon gameplay advice without needing to searc online.

The operative phrase of that comment being “no special part.”

If you watch the Twitch stream it is obvious Claude has general knowledge of what to do to win in Pokémon but cannot recall specifics.

For eg., Bug type attack is super effective against Poison type in Gen 1 but not very effective in Gen 2 and onnwards. But Claude keeps bringing Nidoran into Weedle/Caterpie.

If you're implying that generalization isn't at play because game knowledge shows up in its training data, you can disabuse yourself of that by watching the stream and how it reasons itself out of situations. You can see its chain of thought.

It spends most of its time stuck and reasoning about what it can do. It might throw back to knowledge like "I know Pokemon games can have a ledge system that you can walk off, so I will try to see if this is a ledge" (and it fails and has to think of something else), but it's not like it knows the moment to moment intricacies of the game. It's clearly generalized problem solving.

Reply on news.ycombinator.com