Arc Prize 2024 Winners and Technical Report
arcprize.org
arcprize.org
One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems hasn't changed since ~2019.
But this flipped late summer. AlphaProof and o1 are evidence of this new reality. All frontier AI systems are now incorporating components beyond pure deep learning like program synthesis and program search.
I believe ARC Prize played a role here too. All the winners this year are leveraging new AGI reasoning approaches like deep-learning guided program synthesis, and test-time training/fine-tuning. We'll be seeing a lot more of these in frontier AI systems in coming years.
And I'm proud to say that all the code and papers from this year's winners are now open source!
We're going to keep running this thing annually until its defeated. And we've got ARC-AGI-2 in the works to improve on several of the v1 flaws (more here: https://arcprize.org/blog/arc-prize-2024-winners-technical-r...)
The ARC-AGI community keeps surprising me. From initial launch, through o1 testing, to the final 48 hours when the winning team jumped 10% and both winning papers dropped out of nowhere. I'm incredibly grateful to everyone and we will do our best to steward this attention towards AGI.
We'll be back in 2025!
It is a great unit test for reasoning -- that's fantastic! And maybe it is indeed the best way to test for this -- who knows exactly. But the claim is a little grandiose for what it is, this is somewhat similar to saying that testing on string parity is the One True Test for testing an optimizer's efficiency.
I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious researchers don't take it as seriously as a result. And that's the kind of people that you want to attract to this sort of thing!
I think there is a potentially good future for ARC! But it might struggle to attract some of the kind of talent that you want to work on this problem as a result.
This is fair critique. ARC Prize's 2024 messaging was sharp to break through the noise floor -- ARC has been around since 2019 but most only learned about it this summer. Now that it has garnered awareness, it is no longer useful, and in same cases hurting progress like you point out. The messaging needs to evolve and mature next year to be more neutral/academic.
This is part of the ethical morass of why some more serious researchers aren't touching the benchmark. People are not going to take it seriously if it continues like this!
I have not heard this from anyone that I work with! It would be a curious violation of info theory were this to be the case.
Certainly, some things cannot efficiently be learned from data. This is a case where some other kind of inductive bias or prior is needed (again, from info theory) -- but replacing deep learning entirely would be rather silly.
Part of the reason that a number of researchers don't take the benchmark more seriously is because it's meant to cripple the results. For example, in the name of reducing brute force search, the compute was severely limited! This turned many off to begin with. The general contention as I understand was to let compute be a reasonable amount, but this would not play well with the numbers game. Because if you restrict compute beyond a reasonable point, it makes the numbers artificially low for people who don't know what's going on behind the scenes. And this ends up biasing the results unreasonably to favor the original messaging, (i.e., "We need something other than deep learning.")
If it was structured with a reasonable amount of compute, and instead, time-accuracy gates were used for prizes, it would be much more open. But people do not use it because the game is rigged to begin with!
Unfortunately due to that, plus the consistent goal-post moving of the benchmark is why it's generally not really held with staying power in the research community -- the messaging changes based upon what is convenient for publicity, and there's unfortunately been a history of similar things in the past in the pedigree leading up to the ARC prize itself.
It is not entirely unsalvageable, but there really needs to be a turnaround of how the competition and prize is managed in order to win back people's trust. Placing a thumb on the scales to confirm a prior bias/previous messaging may work for a little while, but over time it robs the metric of its usability over time as the greater research community loses trust.
Much of this comes in reference to statements from fchollet w.r.t. replacing deep learning -- around the time of the initial prize, with a lot of the much more hype marketing, this was essentially the thru-line that was used, and it left a bitter taste in a number of peoples' mouths. W.r.t. misquoting, they did say that we needed something "beyond" deep learning, not "other than" here, and that is on me.
The utility is certainly still present, if I feel diminished, and it probably is a case of my own frustrations due to previous similar issues leading up to the ARC prize.
That being said, I do agree in retrospect that my response skewed from being objective -- it is a benchmark with a mixed history, but that doesn't mean that I should get personally caught up in it.
The entire benchmark is set up so as to try and make it _artificially_ hard for deep learning: there are only three examples for each task; AND the private test set has a different distribution than the public training and validation sets (from what I can tell; a violation of PAC-Learning assumptions and then why should anyone be surprised if machine learning approaches in general can't deal with that?).
Even I (long story) find ARC to be unfair in the simplest sense of the word: it does not make for a level playing field that would allow for disparate approaches to machine learning to be compared fairly. Strangely and uniquely, the unfairness is aimed at the dominant approach, deep learning, where every other benchmark tends to skew towards deep learning (e.g. huge feature-based, labelled data).
But why's that? If ARC-AGI is a true test of AGI, or intelligence, or whatever it is supposed to be (an IQ test for AIs) then why does it have to jump through hoops just to defend itself from the dominant approach to AI? If it's a good test for AI, and the dominant approach to AI can't really do AI, then the dominant approach should not be capable of passing the test, without any shenanigans with reduced compute or few examples.
Is the purpose to demonstrate that deep neural nets can't generalise from few examples? That's machine learning 101 (although I guess there's still those who missed the lecture). Is it to encourage deep neural nets to get better at generalising from few examples? Well, first place just went to a big, deep, bad neural net with data augmentation so that doesn't even work.
Even revisiting it now, I don't see anything wrong with being concisely clear and even a little provocative in stating your case on your own site. Especially since a key value of ARC is getting more objectively grounded regarding progress toward AGI. On top of that ARC is "A non-profit for the public advancement of open artificial general intelligence" that you guys are personally donating serious money and time to that's helping a field where a lot of entrepreneurs are going to make money and academics are going to advance their careers.
My perception is ARC tried it the other way for years but a lot of academics and AI pundits ignored or dismissed it without ever meaningfully engaging with it. "Sharpening" the message this year has clearly paid off in bringing attention that's shifted the conversation and is helping advance progress toward AGI in ways nothing else has. I also greatly appreciate the time and care you and Francois have put into making the ARC proposition clear enough for non-technical people to understand. That's hard to do and doesn't happen by accident.
Personally, I've found ARC valuable in the real world outside of academia and domain experts because it provides a conceptually simple starting place to discuss with non-technical people what the term AGI might even mean. My high school-aged daughter asked me about vague AGI impending doom scenarios she heard on TikTok. I had her solve a couple ARC samples and then pointed out that today's best AIs aren't yet close to doing the same. This counter-intuitive revelation got her pondering the "Why?" which led to a deep discussion about the multi-dimensional breadth of human creativity and an appreciation of the many ways artificial intelligences might differ from human intelligence.
Your perception is very wrong and the likely reason is that as you say you're not an academic researcher. ARC made a huge splash with the original Kaggle competition a few years ago and it drew in exactly the kind of "academic researcher" you seem to be pointing to: those in university research groups who do not have access to the data and compute that the big tech companies have, and who can consequently not compete in the usual big data benchmarks that are dominated by Google, OpenAI, Meta, and friends. ARC, with its (unfair) few-shot tasks and constantly changing private test set, is exactly the kind of dataset that that kind of researcher are looking for, something that is relatively safe from big tech deep neural nets. Even the $1 million prize seems specially designed to be just enough to draw in that crowd of not super-rich academics while leaving corporate research groups insufficiently motivated.
Besides which, I won't name names but one of the principal researchers in the winning system is just one of those academics. I don't know which is the period you mean ARC was ignored by the academic community but that particular researcher was in a certain meeting of like-minded academics two years ago where one of the main areas of discussion was in short "how to beat ARC and show that our stuff works".
How so? All the three top systems are deep neural net systems. The first place went to a system that, quoting from the "contributions" section of the paper, employed:
>> An automated data generation methodology that starts with 100-160 program solutions for ARC training tasks, and expands them to make 400k new problems paired with Python solutions
As I pointed out in another comment the top results in ARC have been achieved by ordinary, deep-learning, big-data, memorisation based approaches. You and fchollet (in these comments) try to claim otherwise but I don't understand why.
In fact, no, I understand why. I think fchollet wanted to place ARC as "not just a benchmark", the opposite of what tbalsam is asking for above. The motivation is solid: if we've learned anything in the last twenty-thirty years is that deep neural nets are very capable at beating benchmarks. For any deep neural net model that beats a benchmark though the question remains whether it can do anything else besides. Unfortunately, that is not a question that can be answered by beating yet another benchmark.
And here we are now, and the first place in the current ARC challenge goes to a deep neural net system trained on a synthetically augmented dataset. The right thing to do now would be to scale back the claims about the magickal AGI-IQ test with unicorns, and accept that your benchmark is just not any different than any other previous AI benchmark, that it is not any more informative than any other benchmark, and that a completely different kind of test of artificial intelligence is needed.
There is after all such a thing as scientific integrity. You make a big conjecture, you look at the data, realise that you're wrong, accept it, and move on. For example the authors of GLUE did that (as in SUPERGLUE). The authors of the Winograd Schema Challenge did that. You should follow their examples.
What do you think about limiting the submission size? Kaggle does this sometimes.
With a limit like 0.1-1MB (compressed), you are basically saying: "Give me sample-efficient learning algorithms, not pretrained models."
On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the correct outputs. In other words, in this context, intelligence is sample-efficiency, as I see it.
Compute is limited during inference, and this naturally limits brute-force program search.
But this doesn't prevent one from creating a huge ARC-like dataset ahead of time, like BARC did (but bigger), and training a correspondingly huge NN on it.
Placing a limit on the submission size could foil this kind of brute-force approach though. I wonder if you are considering this for 2025?
In the case of TTT, I wouldn’t really describe that as a ‘new AGI reasoning approach’. People have been fine tuning deep learning models on specific tasks for a long time.
The fundamental instinct driving the creation of ARC - that ‘deep learning cannot do system 2 thinking’, is under threat of being proven wrong very soon. Attempts to define the approaches that are working as somehow not ‘traditional deep learning’ really seem like shifting the goal posts.
The new and surprising thing about test-time training (TTT) is how effective it is an approach to deal with novel abstract reasoning problems like ARC-AGI.
TTT was pioneered by Jack Cole last year and popularized this year by several teams, including this winning paper: https://ekinakyurek.github.io/papers/ttt.pdf
Was it? What did the "public" consist of exactly?
They do mention that some of the tasks here are susceptible to brute force and they plan to address that in ARC-AGI-2.
> nearly half (49%) of the private evaluation set was solved by at least one team during the original 2020 Kaggle competition all of which were using some variant of brute-force program search. This suggests a large fraction of ARC-AGI-1 tasks are susceptible to this kind of method and does not carry much useful signal towards general intelligence.
For instance, current top models use TTT, which is a completely general-purpose technique that provides the most significant boost to DL model's generalization power in recent memory.
The other category of approach that is working well is program synthesis -- if pushed to the extent that it could solve ARC-AGI, the same system could be redeployed to solve arbitrary programming tasks, as well as tasks isomorphic to programming (such as theorem proving).
From a mathematical perspective, this doesn't sound right. All NNs are universal apprxomators and in theory can all learn the same thing to equal ability. It's more about the learning algorithm than the architecture IMO.
I recently explained why I like ARC to a non-technical friend this way: "When an AI solves ARC it won't be proof of AGI. It's the opposite. As long as ARC remains unsolved I'm confident we're not even close to AGI."
For the sake of being provocative, I'd even argue that ARC remaining unsolved is a sign we're not yet making meaningful progress in the right direction. AGI is the top of Everest. ARC is base camp.
o1-preview doesn't even have image input, so I wonder how they used it.
Also, Ryan Greenblatts solution basically does "best of 4000" iirc. Presumably o1-preview was single shot.
So what happened to that assumption? Is it dead?
The second question I still have is about the defenses of ARC against memorisation-based, big-data approaches. I note that the second best system is based on an LLM with "test time training" where the first two steps are:
initial finetuning on similar tasks
auxiliary task format and augmentations
So what happened to the claim that ARC is secure against big-data approaches? Is it dead?
* Use a pretrained LLM and hope that relevant programs will be memorized via exposure to text data (this doesn't work that well)
* Pretrain a LLM on ARC-AGI-like data
* Hardcode the priors into a DSL
> Which is to say, a data augmentation approach
The key bit isn't the data augmentation but the TTT. TTT is a way to lift the #1 issue with DL models: that they cannot recombine their knowledge at test time to adapt to something they haven't seen before (strong generalization). You can argue whether TTT is the right way to achieve this, but there is no doubt that TTT is a major advance in this direction.
The top ARC-AGI models perform well not because they're trained on tons of data, but because they can adapt to novelty at test time (usually via TTT). For instance, if you drop the TTT component you will see that these large models trained on millions of synthetic ARC-AGI tasks drop to <10% accuracy. This demonstrates empirically that ARC-AGI cannot be solved purely via memorization and interpolation.
Now that the current challenge is over, and a successor dataset is in the works, can we see how well the leading LLMs perform against the private test set?
For example, Claude 3.5 gets 14% in semi-private eval vs 21% in public eval. I remember reading an explanation of "semi-private" earlier but cannot find it now.
Do you mean the ones from your white paper? The same ones that humans possess? How do you know this?
>> The key bit isn't the data augmentation but the TTT.
I haven't had the chance to read the papers carefully. Have they done ablation studies? For instance, is the following a guess or is it an empirical result?
>> For instance, if you drop the TTT component you will see that these large models trained on millions of synthetic ARC-AGI tasks drop to <10% accuracy.
That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human mind, a tiny portion of which solves ARC quickly and easily using causality-first principles of reasoning.
Another point: suppose a system solves ARC-AGI with 99% accuracy. Then it should be tested on "HARC-HAGI," a variant that uses hexagons instead of squares. This likely wouldn't trip up a human very much - perhaps a small decrease due to increased surface area for brain farts. But if the AI needs to be retrained on a ton of hexagonal examples, then that AI can't be an AGI candidate.
This isn’t my area of expertise, but it seems plausible to me that what you said is completely erroneous or at the very least completely unverifiable at this point in time. How do you quantify how many bits it takes a human mind to solve one of the ARC problems?
That seems likely beyond the level of insight we have into the structure of cognition and information storage etc etc in wetware. I could of course be wrong and would love to be corrected if so! You mentioned a “tiny portion” of the human mind, but (as far as I’m aware), any given “small” part of human cognition still involves huge amounts of complexity and compute.
Maybe you are saying that the high level decision making a human goes through when solving can be represented with a relatively small number of pieces of information/logical operations (as opposed to a much lower level notion closer to the wetware of the quantity of information) but then it seems unfair to compare to the low level equivalent (weights & biases, FLOPs etc) in the ML system when there may be higher order equivalents.
I do appreciate the general notion of wanting to normalize against something though, and some notion of information seems like a reasonable choice, but practically out of our reach. Maybe something like peak power or total energy consumption would be a more reasonable choice, which we can at least get a lower and upper bounds on in the human case (metabolic rates are pretty well studied, and even if we don’t have a good idea of how much energy is involved in completing cognitive tasks we can at least get bounds for running the entire system in that period of time) and close to a precise value in the ML case.
These AI output as tokens hundreds of potential solutions, whereas a human solving a very tricky ARC problem might need at most a few dozen cases to run through. There's a big mess of ANN linear algebra / human subconscious thought and I agree these messes can't be compared (or even identified in the human case). But we can compare the efficiency of the solution. It is possible that subconsciously humans "generate" hundreds of solutions that are mostly discarded, but I don't think the brain is fast enough to do that at the speed of conscious thought: it's a 50bn core processor but each core is only 200Hz and they aren't general-purpose CPUs. It also seems inconsistent with how humans solve these problems.
I believe energy usage would be even more misleading: in terms of operations/second a human brain is comparable to a 2020s supercomputer running at 30MW, but it only consumes 300 watts. (I was thinking about this with the "tiny portion" comment but it is irrelevant.)
I guess I’m not sure it’s obvious where the right line to draw the boundary for “intentional human thought” is? Surely there is a lot of cognition and representation going on at extraordinary speeds that exist in some hazy border region between instinct/reflex/subconscious and conscious thought. Still, having said that, I do see what you are saying about trying to compare the complexity of the formal path to the solution, or at least what the human thinks their formal path was.
I’m generally of the mind (also, ha) that we won’t really ever be able to quantify any of this in a meaningful way in the short term and if anything which qualifies as AGI does emerge, it might only be something which is an “I know it when I see it” kind of evaluation…
Where are you getting 300W from? The body only dumps 100W of heat at rest and uses like 300-400W during moderate physical activity, so I’m a little confused about what you are describing there. The typical estimates I’ve seen are like 20W or so for the brain.
Edit: I should also say that what you describe does seem like a great way to compare solutions between computational systems currently being developed and a good one to use to try to push development forward; it just seems quixotic to try to be able to use it comparatively with human cognition or to be able to meaningfully use it to define where AGI is, which might not be what you were advocating for at all, in which case, sorry for misinterpreting!
Either I'm really dumb or the test is getting into captcha-like territory where humans aren't really good at solving/deciphering the test anymore.
In https://arcprize.org/play?task=79369cc6 , the yellow 3x3 square shows you the pink pattern to look for, while allowing rotations (and ignoring the fact that the pattern may be next to other patterns)
1. Existing brute force algorithms solve 40% of this "reasoning" and "generalization" test.
2. AGI must evidently fit on a single 16GB, decade-old GPU?
3. If ARC fails blind people, it's not a reasoning test. Reasoning is independent of visual acuity. So ARC is at best a vision processing then reasoning test. SotA model "failure" is meaningless. ("But what about the other format, JSON?" Yeah, I would love to see the human solve rate on that...)
If it's so easy to brute force, why wouldn't you claim the $1M?