- Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat.
- BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM.
As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these challenges.
This is not AGI at all.
My main criticism would be that it doesn’t seem like this test allows online learning, which is what humans do (over the scale of days to years). So in practice it may still collapse to what you point out, but not because the task is unsuited to showing AGI.
This has absolutely nothing to do with AGI. Once they beat these tests, new ones will pop up. They'll beat those, and people will invent the next batch.
The way I see it, the true formula for AGI is: [Brain] + [External Sensors] (World Receptors) + [Internal State Sensors] + [Survival Function] + [Memory].
I won't dive too deep into how each of these components has its own distinct traits and is deeply intertwined with the others (especially the survival function and memory). But on a fundamental level, my point is that we are not going to squeeze AGI out of LLMs just by throwing more tests and training cycles at them.
These current benchmarks aren't bringing us any closer to AGI. They merely prove that we've found a new layer of tasks that we simply haven't figured out how to train LLMs on yet.
P.S. A 2-year-old child is already an AGI in terms of its functional makeup and internal interaction architecture, even though they are far less equipped for survival than a kitten. The path to AGI isn't just endless task training—it's a shift toward a fundamentally different decision-making architecture.
that's exactly the point! once we cannot invent the next batch (that is easy for humans to solve), that will be AGI
The item I think you should add is a Mesolimbic System (Reward / Motivation). I think AGI needs motivation to direct its learning and tasks.
Also, I don't think the industry has just been training LLMs with more data to get advancement the last 2 years. RAG / Agents loops / skills / context mgmt are all just early forms a Memory system. An LLM with an updatable working set memory is a lot more capable than just an LLM.
Well, the "G" in AGI is kinda important. These are specifically games/puzzles.
> they have to be retrained from scratch
Is that true? Didn't DeepMind already build plenty of agents that are generally good at most computer games without being retrained?
I've been a gamer for just about 40 years. Gaming is my "thing"
I found the challenges fun, but easy. Coming back and reading comments from people struggling with the games, my first thought was - yup definitely not a gamer.
My approach was to poke at the controls to suss the rules, then the actual solutions were really straightforward.
fwiw, I'm pretty dumb generally, but these kinds of puzzles are my jam.