Don't be fooled–LLMs don't reason
technologyreview.com
technologyreview.com
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning?
As:
Do we have evidence humans have an independent set of beliefs not shaped by observation and experience?
Because, that will be a priori or pure reason.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
Models are now starting to show emergent behavior that they were not explicitly trained for. This is leading to the current debate.
The interplay between deductive frameworks and inductive frameworks is still always buoyed by the black swan problem… which is a the heart of Hume’s solipsism. Kant’s pure reason never considered instinctual logical gates that come prepackaged in the mind. That seems frighteningly close to a prediction algorithm, no matter how much we wrap it in a concept of understanding. In the end, we’re just building a model with a minimal error rate.
Based on what? Just because you can model the weather using math doesn't imply at all that the weather is the product of a mathematical model itself. I don't see how reasoning is different.
We deliberately construct LLMs so that they approximate certain relations typical of reasoning. Suppose they become extraordinarily successful at doing so. What exactly have we demonstrated? That these effects can be artificially reproduced through such a mechanism. But how would it follow that the thing being reproduced must itself be nothing more than that mechanism?
Based on the face that we have zero explanation for consciousness. Based on the face that biologically neural networks are exactly a kind of neural network.
My point isn’t that we’re able to explain consciousness as an illusion. It’s that we’re able to explain consciousness-like behavior presumably without consciousness, and the infrastructure look uncomfortably like we do.
Which tradition are we to follow then whilst training the AI models? Pushed beyond their limits today's models tend to hallucinate and cannot be accepted sensu proprio. Philosophers will call it speculative reasoning. And yet, some speculation can be useful in breaking new grounds. Anyway, I have no answers. Just musings.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
You cannot have a syllogism without propositions.
In such systems, a belief is a proposition you accept as true. (provisionally at least)
For example as a scientist, you probably use the epistemology called empiricism all day. In empiricism, beliefs are justified by observation.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
I hope you agree that this question is too important to leave to gut feelings.
Humans have a poor track record in that regard.
This is a simple and intentional design choice though. Biological lifeforms are always on and always receiving sensory input. LLMs don't functionally exist out side of when we decide to run them. An always on agent with a looping prompt of "if you aren't doing anything else, ruminate" overcomes this limitation.
In fact, if your brain were removed from your body, and you were locked in a room that was completely empty and precisely calibrated to your brain's ambient temperature, and you were then ordered on your life to not ruminate. Well -- I would posit that you wouldn't last very long.
There's something else going on with human cognition -- and reasoning by extension -- that LLMs are, to date, not replicating.
With the big caveat of AFAIK. I'm not in one of the labs close to this stuff.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
[1] https://www.researchgate.net/publication/229060046_Telling_m...
[2] https://pubmed.ncbi.nlm.nih.gov/16210542/
“So convenient a thing to be a reasonable creature, since it enables one to find or make a reason for every thing one has a mind to do.”
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
It is possible.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
Just because I can't count every grain of sand on Earth, doesn't mean sand grains are uncountable. Those are two distinct facts.
The practicality of a binary brain is distinct from the possibility of it.
Genuinely I don't understand this.
If the brain isn't just a physical system, then we may as well give up. There's certainly no point discussing it, as any assertions will be untestable and one persons elaborate and well thought out theory will be just as valid and predictive as the next persons "consciousness is created by invisible purple unicorns" theory.
So if we are going to discuss it, we should start from the assumption that there is no magic or witchcraft or religion involved.
Because we've never in the history of science discovered a non-physical (supernatural) system. So it would be quite a surprise.
Oh the irony of you talking about believing in magic. What do you base this wild claim on? Can you cite a single known physicist or mathematician that agrees?
> And so in its actual procedure physics studies not these inscrutable qualities, but pointer-readings which we can observe. The readings, it is true, reflect the fluctuations of the world-qualities; but our exact knowledge is of the readings, not of the qualities. The former have as much resemblance to the latter as a telephone number has to a subscriber.
-- Arthur Stanley Eddington
You don't even need to believe them, just think about how you would ever be sure you know objective reality fully. Actually do it.
When I was a little kid, my go to thought experiment was imagining atoms as balls that we can't crack open that are filled with sand, and that even if we could find the formulas that describe their movements perfectly, we could never know if that's how atoms move, or if there's something inside them (the sand). Really dumb, but I was like 9 yo and knew nothing, never heard of Gödel or quantum mechanics (what's your excuse?). But I still could understand, not intuitively, but by actually thinking (not just talking) about it, that you can never be 100% sure from inside. Even if you found out everything, and your model of the world matches it perfectly, you could never be sure that it is so. And it turns out this is true and an old discovery. If you could disprove it, you would be famous for millenia, maybe forever. Bluffing on HN won't get you there.
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
And... I remember (all of six years ago) back when language was considered the pinnacle of the human mind. Sure, animals might be smart, but they don't have language!
The moment LLMs appeared it suddenly took a back seat to physical navigation and child rearing.
Feels like more gap seeking to me. But time will tell.
My position is that it is not possible.
Seems so to me.
So explain then what you mean by "not possible".
So what about reasoning is not computable in your mind?
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
Love to see the forever moving goalposts. One might say they're autonomously moving..
Do we have finally reliable autonomous driving? And available outside the bay area or whatever small place compared to the rest of the world?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
When I stumble around trying to explain my thinking though I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't? The only response I have is that when the coin doesn't appear you're going to be putting yourself way further behind the starting line then if you thought from the get go that there's no such thing as magic.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
forgive typos, i am on mobile.
https://www.finextra.com/blogposting/31255/adaptability-as-e...
Jacobians are essentially derivatives but for matrices.
Textise is similar and also doubles as a poor man’s text-only browser. https://www.textise.net/referrer
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
(I'm definitely one of the bigger "AI" skeptics out there but am nonetheless fascinated by these questions).
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.
I think the real question is: how do we define reasoning?
My view is that AI can reason, just differently from humans.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
At least reasoning is not guaranteed.
Would sure be nice though...
> First, these models typically maintain no explicit, persistent, and inspectable epistemic state... there is no independent, explicitly represented set of beliefs.
We cannot decipher it, but Mechanistic Interpretability research does show that models are applying and manipulating abstract concepts and relationships encoded in the weights to derive their responses. We can even identify and manipulate those weights, see e.g. Golden Gate Claude. I assume these are the "independent set of beliefs" and the only reason they are not "explicitly represented" is that the representation is too complicated for us to decode.
In fact, this is also how we know they "concoct" chains of thought, because their reasoning traces do not always align with what is going on in their weights (i.e. the "concepts" that were activated during inference.) Lookup chain of thought faithfulness research, something TFA directly cites.
As another comment (https://news.ycombinator.com/item?id=49934166) points out, there is very robustly replicated evidence that humans do something similar (lookup "post-hoc rationalization.") So it is extremely fascinating that LLMs do the same thing!
Can't help but wonder if that's an entirely unrelated though similar-looking phenomenon, or an emergent property of intelligence, or something transmitted subliminally via training on the data our brains we produced...