This really hit me some time back when I was explaining AI to a friend. After about 10 mins of rambling about LLMs and mentioning the attention paper like I knew what I was talking about, it ended with “oh so it’s just a really advanced auto correct”
You could say the (nearly) same thing about search. And content recommendation. And clustering. And topic modeling. And outlier detection. And spam filtering. And image diffusion. And dimension reduction. And...
There's a lot in common between these things, but there's also a lot cool and different!
For transformers in particular, it's pretty cool that you get some WILD emergent properties simply from scaling up.
So yes, it's just a next token predictor, but I'm just a bundle of nerves and meat. I don't get a lot out of those descriptions.
> They're Made out of Meat
https://web.mit.edu/people/dpolicar/writing/prose/text/think...
I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).
If you're an LLM trying to symbolically emulate human behavior, especially when coding, you're going to see git commits, where people write a bunch of code and then go, oh wait, I should fix that!
From the perspective of an outside observer, making mistakes, and then correcting the mistakes, is how you code.
In fact, an outside observer might think that this is how you teach people to code. You make a mistake, then you say, oh, here's the mistake I made, and here's why, and then you fix it.
So I think true emergent behavior will be demonstrated, when an LLM says "Hey wait a minute! They're actually making real mistakes, and they're actually correcting, it's them not on purpose?!", and then stops hallucinating, and stops making mistakes as it codes.
The same is true of bug trackers. You submit a bug request and you often have a patch. Then people discuss it. From a MLM's perspective, you're supposed to write poor code, or code that's not perfect, and then have a conversation.
If I had the resources to train a large-scale LLM, I would clone, for example, GitHub. I would then remove everything where people are fixing broken code.
The outcome would be very interesting.
I've been looking at ways to make enhanced long-term memory stores for LLMs, and there's lots of problems with shifting symbolic relationships if you do it wrong, but definitely once there's true long-term memory, and adaptive behavio,r I think that's the only way you're going to get true real emergent behavior.
I think when people are irritated by the "hallucination" aspect of LLMs, they are often running into something of a slightly different nature. I mean, firstly, there seems to be a higher-than-normal ratio of these "little mistakes". And secondly, there are some pretty odd ones - e.g. in my team, Claude regularly just straight up makes up Jira ticket numbers, and then refers to them with high confidence. I guess what I'm saying is that a human would probably not just make up an id, and run with it (they may be off by one, or mix two up, etc). In my opinion, these can be successfully treated, but I guess you can never fully eliminate the tendency that irks people.
In this specific instance I mean I don't think it was obvious given the structure of the model, it only "emerged" when we applied massive scale, which allowed for unexpected interactions to happen in the deep networks involved.
To me the heart of the "next token predictor" is that the distributions are static. You can manipulate what you feed into it through context (and a lot of interesting engineering has been applied there through CoT and other techniques to manipulate the prompt). But these models as implemented will never be able try things and learn from mistakes or adapt. They are a set of weights frozen in time. A set of distributions derived from the original data that created them.
And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.
What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base model than incorporating new knowledge (at least from my understanding, I am obviously not a researcher at a lab).
My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction.
But as an assessment of the current state, I agree with you, LLMs lag humans severely in ability to self-update.
"LLM" is a branded model as a product. Of course it could be anything, as long as it fulfills the product category.
But we live in reality, we can only look at what models are out there and we see that they don't do any of those things and yet we're supposed to act as if these models already do.
What can a transformer not do that people say they can do?
The parent comment said, paraphrasing, "learn from interaction with the world", and I'm responding, they absolutely can already do this by taking their logs of interaction with humans and updating their weights through backprop.
The reason you don't see that done "live" is primarily an economics problem rather than a limitation of the model structure.
Because backprop fundamentally wants the entire data set in every pass. It doesn't behave well and is destructive when you update after the pre-training phase. RLHF/LORA are attempts to work around that and effective at what they do, but it is not learning in the sense you are talking about and also do not fully address the catastrophic forgetting problem. This architecture as is - is not compatible with continual learning.
It IS done with every new model release. Why do you think SpaceXAI bought Cursor? And then immediately had a huge jump in capability with their next model version?
It's just done in large batches for economic reasons.
> backprop fundamentally wants the entire data set in every pass
I'm genuinely unsure what you mean, it's not even possible to run backprop in this way?
Bolting on new data to an existing model (fine-tuning) is precisely what gets you catastrophic forgetting.
1. Technical cost of updating the mode.
2. Inability to trust every user's "truth".
3. Ability of AGENT-HARNESSES to learn with the help of the human user.
So agents learn, LLM already knows everything it will ever know, and ESPECIALLY it has already learned how to understand human language.
No 3. above means there is no danger of the LLM getting corrupted. But the agents running on user's machine learn on behalf of that user who shares the machine with them.
RNNs and Mamaba do not update their weights, but you could hypothetically scale the internal state to be as big as Fable's and GPT 6's parameters.
It just has big technical and economic challenges. But I expect advances there. There have actually already been big advances, though done in bulk fashion (RLHF).
Sometimes I feel we lack the words to describe what we're doing in a way that really conveys how all these things Ladder Up