Hacker News Reader

Comment by onlyrealcuzzo | Hacker News Reader

His main LLM predictions have almost nothing to do with Arc AGI...

What exactly was he dead wrong about that is proven by any of this?

GPT getting better has absolutely nothing to do with completely disproving anything LeCun has been saying.

He never said LLMs couldn't get better. He never said they couldn't score 7.6% on Arc AGI 3.

He's merely said they don't think, and you probably want something that actually thinks if you want a model that can be trained cheaply on a small amount of data and provide a ton of value.

Spending $5B to train a model that scores better than an older model does not disprove any of that in any way.

View on HN

>He's merely said they don't think

He said years ago even 'GPT 5000' couldnt do things that they ended up doing fine a month later, let alone by 5000. His later predictions are just moving that goal post including towards them not being able to do more general, harder problems of which Arc AGI is a counter-example.

> He said even 'GPT 5000' couldnt do things that they could do a month later, let alone by 5000.

What things specifically and when?

https://youtube.com/shorts/zQTt8TkcyfU?is=09r7XDqz2w6-Pygu

You probably wont like the edit but I dont have the timestamp of the original on hand, you can find it.

That does not at all look cherry picked or taken out of context...

LeCun's ideas cannot be reduced to a 6 second clip...

You're missing the forrest for the trees, taking a singular example of a problem and thinking that if an LLM can solve the singular example it completely disproves LeCun is comical...

You can watch the whole Lex Friedman interview, it's on youtube. It's not out of context at all. He goes on about how LLMs will never be able to do things that they do trivially. And he has just doubled down for years.

Ive read and watched more of his interviews and lectures it seems, it feels like you just have a rosier idea of his views than the views he repeatedly presents.

I think you misunderstand what he thinks LLMs can and can't do. He says repeatedly that LLMs may be great for generating text and code, but that a fundamental model of artificial intelligence should be able to perceive and use information outside of text. Also, that a supervised learning approach is not exactly representative for how we learn. That it's a small piece but we largely learn with unsupervised learning. His main criticisms of LLMs are that they are supervised, probabilistic, and learn largely from text instead of observations. His claims about performance come downstream and I'd still argue that he's been (somewhat) right about those as well.

That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.

While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.

LLMs dont just use text for a while now. It's also not fully supervised for a while.

LLM pre-training is definitely unsupervised.

I think he argues more that LLMs are a form of supervision because you need a whole batch of text about specifically what you want the LLM to learn for it to be useful at that one topic. That they predict on human supplied data instead of learning purely from existing in the world and from observations.

Either way, there's something fundamentally inefficient about the whole business of training LLMs.

The human brain manages to self-organize with only a fraction of the information that LLMs get trained on. To train an LLM you need a lot of high-quality data.

There are two threads in history: firstly the compute thread leading to GPUs and AlexNet in 2012. Secondly the model architecture thread that started long before we had the compute and lead to transformers in 2017.

If the compute thread had been 30 years behind then we might be spending this century coming up with better architectures to make do with the more limited compute. However since the compute came first, we settled on the first thing that worked (transformers) and all effort went into polishing that.

There's something wrong with transformers though. No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.

>The human brain manages to self-organize with only a fraction of the information that LLMs get trained on.

So? The question isnt can we get to ASI as efficiently as a brain, the question is can we get there, which we likely can. The inefficiencies can also be fixed after that.

>No matter how many trillions of dollars get thrown at the problem, they still don't learn like humans do.

Again, so? Humans are efficient but also bad at many things that transformers are already better at because of it. You are looking at the wrong thing if you think it needs to be like humans.

All of us were shocked with RL on LLMs.

To me LLMs have gotten better since 2024, but their fundamental flaws still seem there.

They hallucinate when it comes to really challenging tasks such as math proofs. They still do not reuse code well and will rewrite functions instead of perusing the standard library.

But this is good news. LLMs are awesome and they are only the first step towards AI being applied everywhere. They are a Model T

It's like scaling a swiss cheese and the holes grow bigger with it. You can't get rid of the holes without making a different cheese.

> What exactly was he dead wrong about that is proven by any of this?

He said as you need more and more tokens models will fall apart because each additional token is a chance for a mistake and they will just exponentially fall apart. But in practice models have learned to identify and self-correct mistakes and if you look at the graphs more inference reasoning tokens almost always give far better accuracy.

That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"

His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).

With JEPA models so far, it seems that PCA on latent vectors suffices.

tldr: embeddings have a lot more room for improvement

One of the big things that enabled extended context to work was training techniques for extending LLMs with reasoning, part of the thing he was saying wouldn't work.

Reply on news.ycombinator.com