Hacker News Reader

Comment by tintor | Hacker News Reader

ARC AGI-3 saturated by Astra! https://arcprize.org/leaderboard

I think it could indicate that "semi-private" dataset likely leaked to their training data.

A dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems.

Still, probably not that much compared to employees targeting it.

It says "Provider Adapter" so presumably they put some manual work in to make this work.

ARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra

tl;dr it's 62% when apples-to-apples to other models, which is still notable.

ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.

Woah, that is a crazy interesting read!

The no-reasoning version scores 35% while the low reasoning one scores 17%? What?

I suspect this is "no reasoning set" which might be "default: medium" or perhaps some smart routing. I don't think it's literally "no reasoning".

It says (none), which usually means reasoning disabled.

I would be surprised if none = use default reasoning

It's simulating the Dunning-Kruger effect.

AGI means that the answers to these benchmarks are now accessible, and the defense capabilities of the testing organizations are now negligible.

Look at those costs!

Right?

Between $18k-40k to run a benchmark.

But scored less on V2 and V1 ... too much overfitting?

https://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance

saturated before (higher degree) AGI-2

Reply on news.ycombinator.com