Are you sure there's a real difference? Do you have a definition of "reasoning" that excludes this?
1. Taking established techniques or concepts and appropriately applying them to novel situations.
2. Inventing or synthesizing new, never-before-seen techniques or concepts
The vast majority of the time, humans do #1. LLMs certainly do this in some contexts as well, as demonstrated by my example above. This to me counts as "understanding" and "thinking". Some people define "understanding" such that it's something only humans can do; to which I respond, I don't care what you call it, it's useful.
Can LLMs do #2? I don't know. They've got such extensive experience that how would you know if they'd invented a technique vs had seen it somewhere?
But I'd venture to argue that most humans never or rarely do #2.
That seems fair, although the distinction between synthesizing something new and combining existing techniques is a bit blurry.
What's missing from LLMs though is really part of 1). If techniques A, B, C & D are all the tools you need to solve a novel problem, then a human has the capability of learning WHEN to use each of these tools, and in what order/combination, to solve that problem - a process of trial and error, generalization and exception, etc. It's not just the techniques (bag of tools) you need, but also the rules (acquired knowledge) of how they can be used to solve different problems.
LLMs aren't able to learn at runtime from their own experience, so the only way they can learn these rules of when to apply given tools (aka reasoning steps) - is by RL training on how they have been successfully used to solve a range of problems in the training data. So, the LLM may have learnt that in specific context it should first apply tool A (generate that reasoning step), etc, etc, but that doesn't help it to solve a novel problem where the same solution step selection process doesn't apply, even if the tools A-D are all it needs (if only it could learn how to apply them to this novel problem).
> Prompt: A man working at some white collar job gets an interview scheduled with an MBA candidate. The man says "I can't interview this candidate, he's my son." How is this possible?
> ChatGPT: Because the interviewer is the candidate’s mother. (The riddle plays on the assumption that the interviewer must be a man.)
This is clearly pattern matching and overfitting to the "doctor riddle" and a good demonstration of how there's no actual reasoning going on. A human would read the prompt and initially demonstrate confusion, which LLMs don't demonstrate because they don't actually reason.
Again, this is just asserting the premise that reasoning cannot include pattern matching, but this has never been justified. What is your definition for "reasoning"?
> This is clearly pattern matching and overfitting to the "doctor riddle" and a good demonstration of how there's no actual reasoning going on.
Not really, no. "Bad reasoning" does not entail "no reasoning". Your conclusion is simply too strong for the evidence available, which is why I'm asking for a rigourous definition of reasoning that doesn't leave room for disagreement about whether pattern matching counts.
But I agree in principle that LLMs can be distilled into large IF/THEN/ELSE trees, that's the lesson of BitNet 1-bit LLMs. The predicate tree being learned from data is the important qualifier for intelligence though.
Edit: in case I wasn't clear, I agree that a specific chain of IF/THEN/ELSE statements in a loop can be generally intelligent. How could it not, specific kinds of these chains are Turing complete after all, so unless you think the brain has some kind of magic, it too is reducible to such a program, in principle. We just haven't yet discovered what kind of chain this is, just like we didn't understand what kind of chain could produce distributed consensus before PAXOS.
It's hard to say whether this has been trained out (it's an old example) or if it's just another hurdle that general model progression has overcome.
Ask someone who has made such a mistake to think a little more on it, and they’ll notice their error. Ask a reasoning model to do literally the same thing, to “think” on it, and it will also notice its error.
If you’re still insist that AI are not reasoning here, then neither are people.
The practicality of the situation is that a lot of problems fall into the second bucket. We all like to think we deal with novel problems, but most of what we can think of was already considered by another human and captured by llm. You had to invent something deliberately unique, and that’s telling. Most startup ideas are invented more than once, for example.
The key shortcoming of the llm is that it is not aware of its own limits. If it ever becomes aware it can outsource such rare things to mechanical Turk.
'This is possible because the man is the candidate's father. When he says "he's my son," he's simply stating their family relationship. The scenario doesn't present any logical contradiction - a father could very well be in a position where he's supposed to interview his own son for a job. This would create a conflict of interest, which is why he's saying he can't conduct the interview. It would be inappropriate and unfair for a parent to interview their own child for a position, so he would need to recuse himself and have someone else handle the interview. The phrasing might initially seem like it's setting up a riddle, but it's actually a straightforward situation about professional ethics and avoiding conflicts of interest in hiring.'
EDIT - this is described better by other posters.
Certainly if a human (or some AI) has learned to predict/reason over some domain, then what they will be doing is pattern matching to determine the generalizations and exceptions that apply in a given context (including a hypothetical context in a what-if reasoning chain), in order to be able to select a next step that worked before.
However, I think what we're really talking about here isn't the mechanics of applying learnt reasoning (context pattern matching), but rather the ability to reason in the general case, which requires the ability to LEARN to solve novel problems, which is what is missing from LLMs.
A system that has a fixed set of (reasoning/prediction) rules, but can't learn new ones for itself, seems better regarded as an expert system. We need to make the distinction between a system that can only apply rules, and one that can actually figure out the rules in the first place.
In terms of my definitions of intelligence and reasoning, based around ability to use past experience to learn to predict, then any system that can't learn from fresh experience doesn't meet that definition.
Of course in humans and other intelligent animals the distinction between past and ongoing experience doesn't apply since they can learn continually and incrementally (something that is lacking from LLMs), so for AI we need to use a different vocabulary, and "expert system" seems the obvious label for something that can use rules, but not discover them for itself.
I don't think it's missing, zero shot prompting is quite successful in many cases. Maybe you find the extent that LLMs can do this to be too limited, but I'm not sure that means they don't reason at all.
> A system that has a fixed set of (reasoning/prediction) rules, but can't learn new ones for itself, seems better regarded as an expert system.
I think expert systems are a lot more limited than LLMs, so I don't agree with that classification. LLMs can generate output that's out of distribution, for instance, which is not something that's classic expert systems can do (even if you think LLM OOD is still limited compared to humans).
I've elaborated in another comment [1] what I think part of the real issue is, and why people keep getting tripped up by saying that pattern matching is not reasoning. I think it's perfectly fine to say that pattern matching is reasoning, but pattern matching has levels of expressive power. First-order pattern matching is limited (and so reasoning is limited), and clearly humans are capable of higher order pattern matching which is Turing complete. Transformers are also Turing complete, and neural networks can learn any function, so it's not a matter of expressive power, in principle.
Aside from issues stemming from tokenization, I think many of these LLM failures are because they aren't trained in higher order pattern matching. Thinking models and the generalization seen from grokking are the first steps on this path, but it's not quite there yet.
How is an LLM going to solve a novel problem with just pattern matching?
Novel means it has never seen it before, maybe doesn't even have the knowledge needed to solve it, so it's not going to be matching any pattern, and even if it did, that would not help if it required a solution different to whatever the pattern match had come from.
Human level reasoning includes ability to learn, so that people can solve novel problems, overcome failures by trial and error, exploration, etc.
So, whatever you are calling "reasoning" isn't human level reasoning, and it's therefore not even clear what you are trying to say? Maybe just that you feel LLMs have room for improvement by better pattern matching?
Higher order pattern matching is Turing complete. Transformers are Turing complete. Memory augmented LLMs are Turing complete. Neural networks can learn to reproduce any function. These have all been proven.
So if computers can be intelligent and can solve novel problems in principle, then LLMs can too if given the right training. If you don't think computers can be intelligent, you have a much higher burden to meet.
> Human level reasoning includes ability to learn, so that people can solve novel problems, overcome failures by trial and error, exploration, etc.
You keep bringing this up as if it's lacking, but basically all existing LLM interfaces provide facilities for memory to store state. Storing progress just isn't an issue if the LLM has the right training. HN has some recent articles about Claude code just being given the task to port some GitHub repos to other programming languages, and they woke up the next morning and it did it autonomously, using issue tracking, progress reports, PRs the hole nine yards. This is frankly not the hard part IMO.
I'm not sure why you want to run a Turing machine emulator on an LLM, when you could just write a massively faster one to run on the computer your LLM is running on, cutting out the middle man, but whatever floats your boat I suppose.
Heck, if you really like emulation and super slow speed then how about implementing Conway's game of Life to run on your LLM Turing machine emulator, and since Life is also Turing complete you could run another Turing machine emulator on that (it's been done), and finally run your AGI on top of that! Woo hoo!
I do think you'll have a challenge prompting your LLM to emulate a Turing machine (they are really not very good at that sort of thing), especially since the prompt/context will also have to do double duty as the Turing machines (infinite length) tape, but no doubt you'll figure it out.
Keep us posted.
I'll be excited to see your AGI program when you write that bit.
And coincidentally, a new paper being discussed on HN is a good example addressing your concern about existing models learning and developing novel things. Here's a GPT model that learned physics just by training on a data:
If the only thing missing to implement AGI was a Turing machine to run it on, then we'd already have AGI running on Conway's game of Life, or perhaps on a Google supercomputer.
> Here's a GPT model that learned physics just by training on a data
It didn't learn at run-time. It was PRE-trained, using SGD on the entire training set, the way that GPT's (Generative PRE-trained Transformers) always are.
In order to learn at run-time, or better yet get rid of the distinction between pre-training and run-time, requires someone to invent (or copy from nature) a new incremental learning algorithm that:
a) Doesn't require retraining on everything it was ever previously trained on, and
b) Doesn't cause it to forget, or inappropriately change, things it had previously learnt
These are easier said than done, which is why we're a decade or so into the "deep learning" revolution, and nothing much has changed other than fine-tuning which is still a bulk data technique.
A rigourous definition of "reasoning" is challenging though, which is why people consistently can't provide a general one that's satisfactory when I ask, and this is why I'm skeptical that pattern matching isn't a big part of it. Arguments that LLMs are "just pattern matching" are thus not persuasive arguments that they are not "reasoning" at some cruder level.
Maybe humans are just higher order pattern matchers and LLMs are only first or second-order pattern matchers. Maybe first-order pattern matching shouldn't count as "reasoning", but should second-order? Third-order? Is there evidence or some proof that LLMs couldn't be trained to be higher order pattern matchers, even in principle?
None of the arguments or evidence I've seen about LLMs and reasoning is rigourous or persuasive on these questions.