Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
schema-harness.github.io
schema-harness.github.io
I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.
they could take open weight model, and check what will be impact from that harness on hold-out
ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests.
It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form.
Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.
No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1.
It took a real algorithmic advancement - reasoning models - to solve it.
https://arxiv.org/abs/2607.06764
The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.
So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.
https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...
"kwargs": {
"max_tokens": 100000,
"stream": true,
"reasoning_effort": "high",
"rate_limit": {
"rate": 2,
"period": 60
}
}
You could only claim that if the numbers of parameters and training tokens remained constant while "reasoning" was added to the base models, which is not the case.
So if you look at the ARC-AGI-1 leaderboard (https://arcprize.org/leaderboard), you can clearly see that the bigger a model the better it performs, and that's for the "reasoning" models, e.g. looking at the graph, Claude Opus 4 is at ~30%, Opus 4.5 is between ~60% and ~80% and Claude 4.7 is at ~90% [1].
Not surprising: LLMs continue to improve in performance as long as more resources are spent to train them. "Algorithmic" advances would show the trend line going the other way, i.e. tokens and parameters decreasing steadily while performance either staying the same or improving.
If you've observed something like that I'll be happy to be corrected but I haven't.
____________________
[1] Incidentally, Opus 4.6 slightly outperforms 4.7 and 4.8 with their -alleged- reduced total parameter count. 4.6 is at 94.0% while 4.7 is at 93.5% and 4.8 at 92.5%.
The point of reasoning models is that some tasks fundamentally require a certain number of serial steps. Base models are limited to learning parallel algorithms because of their parallel training, and so struggle on inherently-serial tasks like solving logic puzzles.
The advancement from reasoning is that it allows LLMs to learn a broader class of algorithms.
The problem with "reasoning" is that for the most part it happens outside of training e.g. as CoT. The base model knows what it knows, it knows what it's trained on, and it can't really go much farther than that.
Now, starting with o3, "reasoning" models are probably (who knows exactly) trained on traces of reasoning, either from automated systems or from human experts, but that doesn't mean they learn any kind of algorithm, just that they learn to reproduce the behaviour of "algorithms" (or of reasoning humans). That can improve performance on certain kinds of task (the ones in the training set) up to a point, but you're not going to make a tiny model perform like one ten times larger just by that.
On the other hand, I do think that LLMs can be made smaller without losing a commensurable amount of performance, like e.g. the Tiny Recursion Model (TRM) which did OK at ARC 1 (45% I think). But then you lose a lot of functionality also. Basically the larger models probably have more parameters than they really need and that's something the industry seems to have realised, but you still need huge parameter counts to reach top performance anyway.
And then there's the training tokens, which aren't getting any fewer.
Small reasoning models do indeed outperform base models that are 10x larger, at these logic/reasoning tasks that require serial computation.
They do not outperform at tasks that rely more on world knowledge or memorization.
In most cases the base model cannot complete logic tasks at all, or only for very small instances; it's reasoning or nothing.
> but that doesn't mean they learn any kind of algorithm
They do indeed learn algorithms and can step through them with CoT. This is what allows reasoning models to, e.g. reliably multiply large numbers by applying the grade-school multiplication algorithm.
So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
This makes sense if the models some how become unified.
That’s called brute force so not really.
Does "don't re-invent the wheel" not apply to agents for some reason?
Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.
And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.
Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
Works fast - tells people how to overthrow the government.
Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.
Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.
Training models seems more akin to raising a kid than a computer application.
In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
Cool to see others trying Baba Is You too.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
> simulator the model builds is comparable to the mental model of the game humans create
Eh, this is kind of sounds like being a prey animal that develops an almost unbeatable colored camouflage and then the predator develops infrared vision making it useless and the prey saying "no fair, you cheated".
People use algorithmic models all the time on problems that are far too difficult or large for their minds to conceptualize, is this not just an extension of that?
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.
As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?
There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.
> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.
This harness does nothing to actually accomplish those goals.
It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.
For instance, is solving a math proof by enumerating all permutations exhaustively on a computer cheating? Does it matter that it is not a proof by construction? That its not descriptive? Of course not. The proof of the four color theorem is all that’s necessary and sufficient to prove it. Calculator at an algebra exam? Who cares. This isn’t an exam, this is the real world. The fact an AI can use a physics harness to perfectly achieve ARC-AGI-3 without attaining those goals demonstrates the power of the technique and that the goals are not necessary for that class of problems. Then find another benchmark that actually demands the goals be necessary and sufficient to achieve the benchmark goals. But don’t denigrate the fact we have technology today that yesterday was a fantasy.
This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.
And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.
The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.
The way I'm reading this isn't that they are writing a game simulator, but rather that they have two things they are evolving - a perceptual model of the game mapping from pixels to objects, and a behavioral model of how each action acts upon these perceptual objects. The behavioral model is written as a program that can be backtested by the game states and actions they have already taken to see if they are correctly predicting the resulting next game state.
The ARC AGI 3 games are non-trivial, and I think it's very impressive to see them doing well using this approach.
I'd agree with their conclusion:
> We read a saturated ARC‑3 as the new beginning: mechanism discovery as a general capability — grounding the causal structure of a world through the agentic loop of action and perception, in environments far richer than a 64×64 grid. This is where we are heading to.
This is the way that an animal learns about it's environment - by observation (and innate biases) to recognize the objects in the environment, and predict their behavior, both autonomous (which AGI ARC 3 doesn't test - the objects in the environment are passive), and in reaction to the animal's behavior. The animal predicts and observes, updating its predictions when it is wrong.
A system that could do this in a messy, dynamic, real-world environment would seem a like genuine step in the direction of animal intelligence, especially if it could ditch the symbolic representations.
on the flip side, the idea that most tests are bad, even standardized tests, the tests that you scored well on that gave you all your opportunities in life: it cuts to the emotional, grounded core, the absolute foundation, of too many people. in the crowd of hacker news commenters; people who buy anthropic shares at retail; the people who work at tech companies; and their kids, families, etc., who are a bunch of nobodies, there are a lot more incentives to believe "stupid fucking arcade games test AGI" than not.
When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
hmm, this is like pass@n until you get the high watermark? How would this mean anything?
Token costs are about the number one thing discussed by businesses these days.
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
This line of thinking terrifies me for the future of humanity...
If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).
Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.
Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.
>> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.
Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?
>> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.
It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.
Have the ARC people (Chollet and friends) claimed that this result is cheating?
Neat. Maybe even deeply interesting. Absolutely garbage write up.