O3 "Arc AGI" Postmortem

I’m as skeptical of OpenAI as just about anyone, but Marcus just seems to engage in the kind of contrarian ranting that happens to people when they spend too much time on social media. I would say a better indication of o3’s performance is François Chollet seeming kind of bummed that LLMs have reached this point, contrary to his expectations.

EDIT:

In the interest of not being overly negative myself, I found Nathan Lambert’s more measured and detailed breakdown far more instructive:

https://www.interconnects.ai/p/openais-o3-the-2024-finale-of...

Yes, there's a lot hype, and also quite a bit of drama, but there's also real progress.

Every time I read anything by Gary Marcus, I feel like we're stuck in a silly loop:

  while True:
    progress = HardWordByMany_function(progress)
    print(summarize(progress))
    >>> Models can now do this previously impossible thing!
    criticism = GaryMarcus_function(progress)
    print(criticism)
    >>> It's not intelligence, because it can't do this other thing.

This is the moment businesses have been waiting for—the game-changer that’s about to redefine everything. Expect to see it rolled out across every industry, in every country, everywhere, by next year. And it’s not just hype; it’s that good. It’s so simple and effective that even the receptionist could deploy it, and the CEO? They can trim down the staff, skip the meetings, and spend their day just asking it questions.

No need for tech experts anymore. Software Engineers? DevOps? Gone. The whole IT department? Practically obsolete. All that’s left is the CEO, the cloud, and this. It’s lean, it’s powerful, and it’s set to shake the foundations of how businesses operate. The future isn’t coming—it’s already here, and it’s going to be wild.

> The video should have been much clearer about what was actually tested and what was actually trained. To the average listener it may have sounded like the AI took the test cold, with a few sample items, like a human would, but that’s not actually what happened.

I have to agree with this. I haven't watched the video, but that's exactly what the hype surrounding it made me think up until I read this sentence. An AI system that can be fine-tuned to achieve human level performance on a variety of intellectual puzzles still sounds like an important advance, don't get me wrong, but being able to manufacture a model that does what you need is very different than having a model that's already capable of everyone's needs.

I usually gag when I read Gary Marcus, but this was somewhat reasonable.

I would love to know what o3 got on ARC-AGI without pre-training.

`flagged` because ?

Summary: "O3 is probably not AGI, despite hot debates on short-form social media sites."