This is not intelligence. It's just a good correlation engine with a very big albeit lossy database of things.
What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.
I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.
That’s a controversial statement.
>What I don't understand is why LLMs haven't been able to do this yet
LLMs are just trained on what humans have said. Why is it surprising that it's still not possible to reconstruct the intelligence that wrote all that by working backwards? Think of your own work experience. When you look at a piece of code, say, are you always able to discern why the person did what they did, just from the code, with no additional context?
Is a fact stored on your brain like digits on a harddrive? No, it's a pathway that lights up and branches when information enters it. It is dynamic, a compressed form you could say, right? The model holds information, but not all information, but enough to be useful (in decision making).
Arguably it's the same, but the model is probably a "compressed" version of the whole fact that took place in reality.
And you can entertain the models internally and sharpen them. Alone or with others.
This doesn't address the only part I really commented on, which is the connection between intelligence and compression.
Asked Google: "Clustering by compression".
I still think this has much more to do with the structure of language than the abstract conception I have of intelligence, and I would be interested in having conversation w/ someone for whom the opposite is true.
The issue isn’t really harness vs. no harness. IMO it’s about the lack of an internally generated sense of what to attend to. Yes, the KV cache accumulates state and its “attention” (if you can even call it that) changes with context. We’ve even managed to /kinda/ close the loop with agentic tool calling and ‘memory’ systems, but these just close the loop at the level of behavior rather than disposition. All agentic harnesses do is make an LLM responsive to the consequences of its actions without changing the tendencies by which it determines what to retain or avoid.
The ghost you can’t escape from at this point is the origin of that relevance. Where does the pull toward one thing mattering over another actually come from? If you ran Fable 5 on a Turing machine and rewound the tape to the exact same state with the exact same input (incl. PRNG seed), it would spit out the same output every time.
Everyone’s trying to outrun this problem by training more often or increasing model sizes. But all this does is inform your model, from the outside(!), what constitutes a better state. The thing that’s actually doing the determining remains unchanged. Congratulations, you’ve scaled the transition function and tape of your Turing machine until it requires every watt generated by ERCOT, and it still cannot, for the life of it, tell you why it should give a shit.
A trained model generating output from weights, a seed, and some context effectively has next-state that’s a total function of those three things. Whatever behavior appears as ‘selecting what is relevant’ is, underneath, just a transition rule executing, no matter how sophisticated or creative the output looks. It can be fully accounted for by what was fixed before it started executing. Which means whatever criterion it uses for determining what matters was inherited from a structure that was already in place before it encountered the situation.
No amount of pruning or post-training can fix this. These approaches just replace one externally supplied criterion with another. For a system to be truly adaptable, there would have to be some criterion by which it treats one possible change as preferable to another, and that criterion itself would have to come from... somewhere. You can even change your conception of ‘improvement’ (e.g. parameter count, harnesses, self-modification, hell, even its ability to spit out shitty best-selling romance novels onto Amazon) and you still haven’t explained where the normative distinction comes from. Every layer of this problem has its root in a preference that was supplied from somewhere else.
I genuinely don’t know if this issue bottoms out anywhere, at least for the way we currently build these systems. Perhaps the solution is still computable, maybe? Who knows what that would even look like. But I’m fairly confident that it isn’t a bigger tape. I hope nobody solves this in the near future because, well, I’d like to have a job...
It's not some tiny "uncomputable spark" you need to look for, most of it is entirely uncomputable.
Where is the computable part in me that is doing all this thinking and being a person? Where is that "tiny spark", point me at it :)
As my AI professor said in the first lecture: “All AI is advanced search”.
Many things are predicted by models in our planet. From weather to production and material science. Building the model needs intelligence, running the model does not.
The person who came up with the formulae for CFD was intelligent. The computer running the model is not. Same for LLMs, chess engines, engine ECUs and financial prediction systems.
Again, for the example’s sake; the person who came up with an algorithm is intelligent. The model mixing its training data to emit something similar is not.
So when LLMs can do all human knowledge work, and do it better than humans, we'll be in the mines listening to you go on about how it's actually just autocomplete or just math, a distinction that apparently means nothing.
No.
> So when LLMs can do all human knowledge work, and do it better than humans, we'll be in the mines listening to you go on about how it's actually just autocomplete or just math, a distinction that apparently means nothing.
With a big "if" attached to it. People were saying "computers will program themselves in the near future" for, checks notes, 24 years now, as far as I'm aware.
We're constantly building new knowledge and understanding things better than olden days. These models just compress our knowledge and light the blind corners we can't see well. I don't say they are useless, but I say that these things are overhyped.
All they can do is regurgitate human knowledge packed into them and highlight some long-distance correlations between items, which is useful in itself, but it can't jump to somewhere where it's not present its training data, but that's something humans and only humans can do.
Most of what you said reads to me as denial.
An unconscious unintelligent but persistent trial and error process created us. We created LLMs. LLMs may create the next thing before we do - hard to say. They don't have all the cognitive tools we have yet, but they still outperform in some areas. As the cognitive playing field levels, I expect you will come to eat your words..
That sounds like something that can be engineered, can't it? In other words, we can identify limitations in current transformer-based architectures, and we can also build new architectures over time.
Briefly, any intelligent creature has internal stochastic processes like sensory inputs and feelings to a certain degree. These stochastic inputs and the creature's own actions change the creature in subtle or profound ways. An LLM has no such processes. You push inputs to the same static model, sans temperature which is just a randomness slider.
Considering the model even doesn't see the words and work on matrices of numbers is even more telling. One needs to add "tools" and other "experts" to overcome the shortcomings caused by this modus operandi.
I can call the algorithm/model smart as in a smartwatch. It can mimic certain things well while having none of the underlying foundation beneath it, or redirect some of the things to correct tools to get deterministic and accurate results if it can't evaluate the query inside its own network in a sane manner.
Coming to your question, "simulating a brain" in a static manner would not make that simulation intelligent, but if you can "wire" it completely and let it evolve by itself, now we're entering a territory I have not spent enough time for thinking it through.
Oh, as I said "I don't know", an LLM doesn't know what it doesn't know, and can't self correct itself which are required capabilities for understanding something. It just generates something statistically viable via its network.
This is because you goal is to state how models are not intelligent, but you couldn't attack the generated text itself, so you created a little rider, attached it to the model, and then you attacked the raider.
But, even in that you failed. You compared the source of human randomness in text generation, and called it 'profound' and implied that it is exactly the source of true intelligence. But, then, the temperature, the similar thing in model was "just a randomness slider". Double standard.
A logical fallacy free attack on LLMs would be to show a prompt, and then the response generated by this prompt, where it would be shown that only an entity with no intelligence would generate such a response. Yet, attacks like this are not written here anymore.
I wonder why.
You point out that I didn't attack the output itself. But the method you propose is deeply flawed.
I can give you n prompts and m results provided by these prompts, all passed through black boxes. And you can't discern the algorithms or models they have gone through. These boxes can range from simple text generators to MATLAB, Mathematica, CFD applications, correlation engines, linear solvers, mathematical proof-checkers, LLMs, you name it.
For any kind of input they can accept, you can't discern whether the algorithm behind it is intelligent or not, because none of the outputs can be produced by something that doesn't pack some kind of smarts.
How do we pack these smarts in? We teach them as intelligent humans. We pack our intelligence inside them as models (aka algorithms). They do a great job of approximating what we know in a smaller, better-designed problem space. We use these approximations to fine-tune our designs or predict things, then go from there. Just because an algorithm is more capable in processing inputs in some cases doesn't make it intelligent. The way the output looks doesn't make the algorithm intelligent, either.
I have developed multi-agent systems which showed emergent intelligence when the agents came together across distributed systems; I have written high-performance modeling software which can do calculations way faster and better than humans in the materials science space. I'm not doing some kind of armchair criticism of what I'm talking about.
> You compared the source of human randomness in text generation, and called it 'profound' and implied that it is exactly the source of true intelligence. But, then, the temperature, the similar thing in model was "just a randomness slider". Double standard.
Nope, my stance is clear. To quote myself:
> Briefly, any intelligent creature has internal stochastic processes like sensory inputs and feelings to a certain degree. These stochastic inputs and the creature's own actions change the creature in subtle or profound ways. An LLM has no such processes. You push inputs to the same static model, sans temperature which is just a randomness slider.
To expand my quote, humans or any living creatures do not stay static. They evolve due to the sensory input they receive from external and internal stimuli. The temperature slider doesn't do anything close to that. You tickle a static model in different amounts. The model doesn't change after you supply the inputs & temperature and get the output. Creatures do not stay the same. Their mood, behaviors, and stance against life and their environment change, sometimes permanently.
I'll go one step further. We are not intelligent enough to understand other living beings around us. Claiming that we can build AGI tomorrow is a god-complex. What we have done is something arguably useful in some cases, but how this is built is another matter which is worthy of its own discussion. However, today I don't have time to re-iterate all the problems over and over. You can search my comments for that, if you are in for it.
So, no. You tried to attack my comment by finding contradictions in it, but you failed. A better rebuttal would try to similarize how LLMs mirror the human learning process and just read like a normal human, but this is a well-trodden path which has been rebutted countless times in various forms.
Nobody is trying to make that claim here anymore.
I wonder why.