The most underreported story in AI is that scaling has failed to produce AGI

I always find the "It's just..." arguments amusing. It presupposes that we know what any intelligence, including our own "is". Human intelligence can just as trivially be reduced down to "it's just a bench of chemical/electrical gradients".

We don't understand how our (or any) intelligence functions, so acting like a next-token predictor can't be "real" intelligence seems overly confident.

I like to describe them as a very powerful tool for quickly creating impressive demos.

“AI” began as a buzzword invented by Marvin Minsky at MIT in grant proposals to justify DoD funding for CS research. It was never equivalent to AGI in meaning.

> which harkened to sci-fi characters like Data or GLaDOS.

There's a truth in there: Today's chatbots literally are characters inside a modern fictional sci-fi story! Some regular code is reading the story, acting out the character's lines, we humans are being tricked into thinking there's a real entity somewhere.

The real LLM is just a Make Document Longer machine. It never talks to anybody, and has no ego, and it sits in back being fed documents that look like movie-scripts. These documents are prepped to contain fictional characters, such as a User (whose lines are text taken unwittingly from a real human) and a Chatbot with incomplete lines.

The Chatbot character is a fiction, because you can simply change its given name to Vegetarian Dracula and suddenly it gains a penchant for driving its fangs into tomatoes.

> The new "reasoning" or "chain of thought" AIs are similarly just a bunch of conventional LLM inputs and outputs stacked on top of each other.

Continuing that framing: They've changed the style of movie script to film noir, where the fictional character is making a parallel track of unvoiced remarks.

While this helps keep the story from going off the rails, it doesn't mean a qualitative leap in any "thinking" going on.

Simple systems layered on top of each other is how we got to human intelligence (presumably).

>AGI", had to be created to represent what "AI" once did.

And, "AGI" has already been downgraded, with "superintelligence" being the new replacement.

"Super-duper" is clearly next.

>a DeepSeek distillation on my PC - where each step of the process is visible - removed quite a bit of the magic behind the curtain.

I always figured that by the time the 1990's came along, there would finally be powerful enough PC's so that an insightful enough individual would eventually be able to use one PC to produce such intelligent behavior that it made that PC orders of magnitude more useful. In a way that no one could deny there was some intelligence there, even if it was not the strongest intelligence. And the closer you looked and became familiar with the underhood processing, the more convinced you became.

And that would be what you then scale, the intelligence itself, even if weak to start with it should definitely be able to get smarter at handling the same limited data if the intelligence was what was scaled more so than the hardware & data.

I'm starting to examine genai products within the framework of a confidence game.

each level above the first is predicting concepts right.

It is far more accurate to say LLMs are collapsing or reducing response probabilities for a given input, than any kind of “thinking” or “reasoning”.

That's adjacent to their point, doing that has given the impression that anthropomorphizing was right, or that things that they do are human like but it is all a facade.

Well that's kind of the problem though, isn't it? All that effort for the machine to sometimes correctly draw the regression line between the right and wrong answers in order to solve a trivial problem. A 6-year old kid would only need to learn the alphabet before being able to count it all on their own. Do we even realise how ridiculous these 'successes' sound? So a machine we have to "train" how to count the letters, is supposed to take over the work which is orders of magnitude more complex? It's a classic solution looking for a problem, if I've ever seen one.

I get the technical challenge. It's just that a system that has to be trained with Petabytes of data, just to (sometimes) correctly solve a problem which a six-seven year old kid is able to solve after learning to spell, may not be the right solution to the problem at hand? Haven't the MBAs been shoving it down our throats that all cost-ineffective solutions have to go? Why are we burning hundreds of billion of dollars into development of tools whose most common use-case (or better said: plea by the VC investors) is a) summarising emails (I am not an idiot who cannot read) b) writing emails (really, I know how to write too, and can do it better) . The only use-case where they are sometimes useful is taking out the boring parts of software development, because of the relatively closed learning context, and as someone who used them for over a year for this, they are not reliable and have to be double-checked, lest you want to introduce more issues in your codebase.

It didn't immunize Eliezer Yudkowsky, and he wrote Markov chain fictional characters. Everyone who looked up AI enough times knew about ELIZA.

One interesting thing that’s shown up in polling (of laypeople) fairly consistently; people tend to become less impressed with, and come to dislike more, LLMs as exposure grows.

To some extent ChatGPT was a magic trick; it really kind of looks like it’s talking to you at first glance. On repeated exposure the cracks start to show.

There's also a perspective that all of the ongoing problems have been the same while newer techniques shove them under different rugs. So I can see how that would look like that to the credulous.

What does it mean when someone is sticking to their guns? Is it a bad thing? I do appreciate consistency, albeit a fair consistency and Gary Marcus's points do stand. When these criticisms are addressed (if it's possible to) you'd probably hear less from Gary Marcus.

Show me the goalposts I have moved, with actual quotes to prove it. nobody ever has when I have asked.

Aso consider eg the bets I have made with Miles Brundage (and offered to Musk(, with money where I have backed up my views.

good summary of predictions i made - mostly correct – is here: https://open.substack.com/pub/garymarcus/p/25-ai-predictions...

How would it be possible to, say, show the reality of a forest fire's devastation while not appearing to show a bias for showing charred trees?

He recently posted a question he put to grok3 — a variation on the trick LLM question (my characterization) of "count the number of this letter in this word." Apparently this Achilles heel is a well-known LLM shortcoming.

Weirdly though, I tried the same example he gave on lmarena and actually got the correct result from grok3, not what Gary got. So I am a little suspicious of his ... methodology?

Since LLMs are not deterministic it's possible we are both right (or were testing different variations on the model?). But there's a righteousness about his glee in finding these faults in LLMs. Never hedging with, "but your results may vary" or "but perhaps they will soon be able to accomplish this."

EDIT: the exact prompt (his typo 'world'): "Can you circle all the consonants in the world Chattanooga"

Garry Marcus constantly repeats the line that "deep learning has hit a wall!1!" - he was saying this pre-ChatGPT even! It's very easy to dunk on him for this.

That said, his willingness to push back against orthodoxy means he's occasionally right. Scaling really has seemed to plateau since GPT-3.5, Hallucinations are still a problem that are perhaps unsolvable under the current paradigm, LLMs do seem to have problems with things far outside their training data.

Basically, while listening to Gary Marcus, you will hear a lot of nonsense, it will probably give you a better picture of reality if you can sort the wheat from the chaff. Listening to only Sam Altman, or other AI Hypelords, you'll think the Singularity is right around the corner. Listen to Gary Marcus, you won't.

Sam Altman has been substantially more correct on average than Gary Marcus, but I believe Marcus is right that the Singularity narrative is bogus.