Comment by pmayrgundter | Hacker News Reader

Comment by pmayrgundter | Hacker News Reader

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC].

In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole might be sufficiently difficult for machines that it does not reveal real progress on machine acquisition of core knowledge."

ConceptARC is designed to be easier, but then also has to filter ~15% of its own test takers for "[failing] at solving two or more minimal tasks... or they provided empty or nonsensical explanations for their solutions"

After this filtering, ConceptARC finds another 10-15% failure rate amongst humans on the main corpus questions, so they're seeing maybe 25-30% unable to solve these simpler questions meant to test for "AGI".

ConceptARC's main results show CG4 scoring well below the filtered humans, which would agree with a [Mensa] test result that its IQ=85.

Chollet and Mitchell could instead stratify their human groups to estimate IQ then compare with the Mensa measures and see if e.g. Claude3@IQ=100 compares with their ARC scores for their average human

[ConceptArc]https://arxiv.org/pdf/2305.07141 [Mensa]https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-10...

Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p...

> We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

(You can direct link to a task like this: https://arcprize.org/play?task=009d5c81 in case you want to share!)

It's pretty easy, just follow the second example with the colors from the test input. (if it's the same puzzle 00576224 for you too)

https://arcprize.org/play?task=00576224 Yes the same puzzle.

And I followed the second example. This was my solution:

GRG

OBO

RGR

B is the cyan like blue color. My solution looks right, but it says it’s wrong.

You need to resize the output grid to 6x6.

How come? The pattern should work for any size grid.

You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color?

The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output.

In short: you need to resize the grid because that’s what the examples do.

> why not just make the grid 1x1 and select a single color?

For two reasons:

1. The initially suggested grid size was 3x3.

2. Filling in a 3x3 grid is sufficient to show that you understood the pattern, but filling in a 1x1 (or even 2x2) grid is insufficient.

Requiring the user fill in a larger grid is a waste of time. The existence of the grid size selector would still make sense in cases where a 2x2 grid would be sufficient to show the solution, so it is not obvious at all that a 6x6 grid should be chosen.

> The grid size is part of the pattern in the same way that the colors are part of the pattern.

To understand a pattern, you have to see at least two valid inputs and corresponding outputs. For the first example, a valid example for the expected output grid size is missing.

I arrived at the "correct" conclusion eventually, but the only indicator was that the reading direction for the UI was absolutely ridiculous ( https://i.imgur.com/CuQ2z2N.png ), suggesting that the authors did not think this through properly, so the solution had to be weird as well.

The fact that two intelligent beings are debating what the correct answer is shows that there is no fixed correct answer that proves "intelligence".

This is IQ tests all over again. Actually testing how alike you think to the author of the test.

Twist: bigyikes is an LLM!

Honestly I’d disagree. I was a bit confused at first but moment I realized I could resize the grid, the answer strikes me as obvious and clear. Yes, in some theoretic sense you can argue a 3 x 3 grid answer is fine, but shows this to 100 different humans and majority would agree that resizing the grid is the obvious and more natural solution.

So anyone who disagrees, including you for a short time, is actually not an intelligent being?

Not to mention that ignoring the size of the grid, one might disagree about the answer of one of the tests.

What is even the meaning of "correct" in this case?

This makes me think of "math" problems requiring you to find the next number in a series. They give you 5 numbers, and ask for the 6th. When I can build a polynomial than can generate the first 5 and any 6th number. Any.

Sounds like the point of these exercises it to guess what the author had in mind, more than some universal intelligence test. Though of course the author thinks their own thoughts are the measure of universal intelligence. It's a tempting thing to believe.

Yeah into the same problem but managed to find the solution by resizing it.

The pattern changes in the middle as well. so you'd need to show on the full size

They claim that the average score for humans is between 85% and 100%, so I think there's a disagreement on whether the test is actually too hard. Taking them at their word, if no existing model can score even half what the average human can, the test is certainly measuring some kind of significant difference.

I guess there might be a disagreement of whether the problems in ARC are a representative sample of all of the possible abstract programs which could be synthesized, but then again most LLMs are also trained on human data.

The tasks are very easy for humans. Out of the 6 tasks assigned when I opened the web page, I got all of them correct on the first try.

Maybe if you run into some exceptionally difficult tasks it might not be 100%, but there's no way the challenge can be called unfair because it's too difficult for humans too.

I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.

Are you working on (a book detailing) AGI also? It’s a lonely field but I have no doubt there are a sea of malcontent engineers across the world who saw the truth early on and are pushing solo for AGI. It’s going well for me, but I’m not sure whether to take that as “you’re great” or “it’s really that easy”, so was interested to see such a fellow brazen American on HN of all places.

Game on for the million, if so :). If not, apologies for distracting from the good fight for OSS/noncorp devs!

E: it occurred to me on the drive home how easily we (engineers) can fall into competitiveness, even when we’ve all read the thinkpieces about why an AI Race would/will be/is incredibly dangerous. Maybe not “game on”, perhaps… “god I hope it’s impossible but best of luck anyway to both of us”?

You actually think that has not been going for 30, 40 or 50 years?

Melanie is coauthor/supervisor of ConceptARC, that can be tried here: https://neoneye.github.io/arc/?dataset=ConceptARC

Reply on news.ycombinator.com