Comment by YeGoblynQueenne | Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

Parent Full thread

YeGoblynQueenne

ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome.

ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests.

It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form.

Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.

Benchmarks are meant to measure something, so can't be too hard else all measurements will be 0. At the same time the systems being tested - LLMs - are getting larger and more capable, at least in the narrow areas most benchmarks are focusing on, so all benchmarks will continually become saturated and need to be revised.

So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.

>It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation

No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1.

It took a real algorithmic advancement - reasoning models - to solve it.

DeepSeek V3.2 was tried without reasoning and it got 57% on ARC AGI 1. It's a 7 month model, so I'm pretty confident that base LLMs would be able to solve ARC AGI 1 without reasoning/CoT.

No, you are misunderstanding the paper.

https://arxiv.org/abs/2607.06764

The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.

I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard. Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score".

So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.

I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category.

https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...

  "kwargs": {
     "max_tokens": 100000,
     "stream": true,
     "reasoning_effort": "high",
     "rate_limit": {
        "rate": 2,
        "period": 60
     }
  }

The other paper ran Deepseek v3.2 without reasoning as a baseline and got 15.5%, which is much more in line with other base LLMs like GPT-5.2.

Interesting, good find! Yeah I may be wrong and this may be an error in the leaderboard. Weirdly it shows no reasoning cost and no reasoning tokens used, but for example here https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... the answer is super short but it says "4945" completion tokens.

>> It took a real algorithmic advancement - reasoning models - to solve it.

You could only claim that if the numbers of parameters and training tokens remained constant while "reasoning" was added to the base models, which is not the case.

So if you look at the ARC-AGI-1 leaderboard (https://arcprize.org/leaderboard), you can clearly see that the bigger a model the better it performs, and that's for the "reasoning" models, e.g. looking at the graph, Claude Opus 4 is at ~30%, Opus 4.5 is between ~60% and ~80% and Claude 4.7 is at ~90% [1].

Not surprising: LLMs continue to improve in performance as long as more resources are spent to train them. "Algorithmic" advances would show the trend line going the other way, i.e. tokens and parameters decreasing steadily while performance either staying the same or improving.

If you've observed something like that I'll be happy to be corrected but I haven't.

____________________

[1] Incidentally, Opus 4.6 slightly outperforms 4.7 and 4.8 with their -alleged- reduced total parameter count. 4.6 is at 94.0% while 4.7 is at 93.5% and 4.8 at 92.5%.

I don't agree with your definition.

The point of reasoning models is that some tasks fundamentally require a certain number of serial steps. Base models are limited to learning parallel algorithms because of their parallel training, and so struggle on inherently-serial tasks like solving logic puzzles.

The advancement from reasoning is that it allows LLMs to learn a broader class of algorithms.

Sorry, which definition do you mean?

The problem with "reasoning" is that for the most part it happens outside of training e.g. as CoT. The base model knows what it knows, it knows what it's trained on, and it can't really go much farther than that.

Now, starting with o3, "reasoning" models are probably (who knows exactly) trained on traces of reasoning, either from automated systems or from human experts, but that doesn't mean they learn any kind of algorithm, just that they learn to reproduce the behaviour of "algorithms" (or of reasoning humans). That can improve performance on certain kinds of task (the ones in the training set) up to a point, but you're not going to make a tiny model perform like one ten times larger just by that.

On the other hand, I do think that LLMs can be made smaller without losing a commensurable amount of performance, like e.g. the Tiny Recursion Model (TRM) which did OK at ARC 1 (45% I think). But then you lose a lot of functionality also. Basically the larger models probably have more parameters than they really need and that's something the industry seems to have realised, but you still need huge parameter counts to reach top performance anyway.

And then there's the training tokens, which aren't getting any fewer.

>but you're not going to make a tiny model perform like one ten times larger just by that.

Small reasoning models do indeed outperform base models that are 10x larger, at these logic/reasoning tasks that require serial computation.

They do not outperform at tasks that rely more on world knowledge or memorization.

In most cases the base model cannot complete logic tasks at all, or only for very small instances; it's reasoning or nothing.

> but that doesn't mean they learn any kind of algorithm

They do indeed learn algorithms and can step through them with CoT. This is what allows reasoning models to, e.g. reliably multiply large numbers by applying the grade-school multiplication algorithm.

Reply on news.ycombinator.com