Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
This line of thinking terrifies me for the future of humanity...
If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).
Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.
Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.
>> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.
Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?
>> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.
It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.
Have the ARC people (Chollet and friends) claimed that this result is cheating?