Can large language models reason?

OTOH I have found that it not as good on topics that there is not a lot of prior art in the public domain or in topics like advanced mathematics which often have a formal and unintuitive presentation.

For example, I have acquired recently an interest in probabilistic logic and when discussing results with it, I have found that although it has knowledge of the subject, it cannot really apply that knowledge in a creative way. It will often make logical mistakes and when these are pointed out, it will first apologize and then continue to make the same mistake.

So as far as I am concerned I am unconvinced that it can reason yet.

But just ask ChatGPT (4o) how a farmer and a sheep would cross a river on a boat. For certain formulations, it reliably hallucinates a wolf or a piece of cabbage into the problem. Why? Because it pattern matches the solution to a well-known puzzle.

We've now seen enough failure modes of these models to realize that a lot of the early successes in logic tests were us not realizing how deep the corpus actually is.

Basically any question humans have ever asked or thought of is in the training data. And ChatGPT has remembered so much of it that it's very hard to surprise the model. But when we manage to do, we see it's actually dumber than a three-year-old. (Which to be fair, is actually still an accomplishment that just a few years ago almost nobody would have considered possible.)

Can an LLM produce output that matches the written aspect of what humans call reasoning? Obviously yes. Are there limits? Yes, but individual humans have limits too.

This explanation pretty much tracks exactly with model performance on reasoning tasks.

Think about how you reason. Even if you write up the results of a problem you're trying to solve, you don't write down every granular mental step you took to get there, or every line of thought you considered and then abandoned.

When the limits of attention have been reached, the model fails to show better performance even with chain of thought prompting. Either the model needs to repeat layers recurrently or it needs an affordance model to understand the relative difficulty of a problem and split it into sub problems that it can actually solve.

I.e. you can't tell if a result was produced by "reasoning" or by a simple lookup.

LOL, no.

My calculator can add numbers to get the same answers I do. I use my intelligence to get those answers, so my calculator must also be intelligent? That's quite an extension you've got there. LLMs mimic intelligence by grinding up everything we feed them and spitting back the patterns they have formulated from that data. They have no ability to extrapolate, as we see demonstrated every day when people report the answers to prompts where the LLM spouts self-evident nonsense.

Well said. People confuse a simple task definition (token continuation), with a simple task, and simple solutions.

But there is no limit to how complex the patterns in data for a continuation task can be.

And there is no limit to the complexity and power that a solution must implement to perform that task.

So these models are not "just" token continuation engines. They are human conversation models, capturing much of the complexity of human thought and knowledge contained in the example conversations.

--

Another factor I don't see given enough weight. The models are currently operating with severe restrictions compared to humans:

1. only given an architecture allowing a limited number of steps (before incoherence).

2. forced to continually output tokens without any "hidden" steps for contemplation, not even initial contemplative steps.

3. required to perform all reasoning within their limited in-line memory, as apposed to being able to use any side-memory such as (the digital version of) note pads or white board.

4. required to generate one answer, whole and coherent, without option to post edit.

5. required to be aware and familiar with a vast cross section of human knowledge, far beyond any single person's fluency, and mix that knowledge sensibly. Languages, subjects, modes of thought, communication and roles.

6. Limited modes of information, i.e. text. (A limit which is now being removed for many models.)

Within those incredible restrictions, they are already vastly better at "reasoning" than a human being being asked to perform the same task with the same restrictions.

--

So at this point, they reason better than us, within those limits. None of us, even experts in an area being tested, would avoid making numerous simple mistakes with those restrictions.

And outside those limitations, they just don't operate yet - so our reasoning is mixed, but often supreme. I say mixed, because no human has familiarity with so much information, or fluency in so many modes of communication, as these models have now - even given a day or two to respond.

Writing a supreme court brief on the implications to a mathematical theorem to the economic impact of some political event in five languages, in normal speech, song, pig latin, Dr. Seuss prose, and as a James Bond story respectively, all in the perspective of a version of the US run by a German federal system in an alternate world where World War II came down differently, isn't something any human being could do as well, no matter how imperfect current models responses to that request might be.

So we are already in territory where they are vastly better at reasoning than us, within clear constraints. And in some cases, better than us outside those constraints, even if those cases might be contrived.

--

That is all verifiable, even obvious.

As far as opinion, I don't see any indication that as those limitations are overcome they won't continue to vastly exceed us - within whatever limitations still exist.

Our brains are amazing. But its worth noting we are not able to scale in terms of training intensity, data quantity, computational power, precise and unwavering memory cells, etc. And our cells have enormous demands on them that transistors don't: maintaining their own structure, metabolism, fighting off chemical and biological adversaries, etc.

Models also train single mindedly. Human learning is always within our biological context - we continually subdivide our attention in real time, in order to remain situationally aware of threats, needs and our social environment. And our brains enforce time limits on our best performance, due to the real time need to remetabolize basic means of operation (neurochemicals), and expectations of limited energy resources.

And models amortize the training of just one model instance, across the inference of any number of copies of that model. They never get tired. Efficiency savings vs. humans that are unprecedented.

I find most pessimistic dismissive arguments are based on criticism against some imagined standard of perfection, instead of a standard of what humans are really capable of when performing the same task with the same limitations.

Humans are notoriously unreliable.

(Some criticisms are valid: such as models current greater propensity to confabulate what they don't know, vs. humans having more awareness of what we remember vs. when we unconsciously fill in the blanks of our memories - but humans do that too. Ask any detective who takes witness testimony.)

That's a very unhelpful metaphor in a sea of misleading narrative. That narrative started the moment we called it "an AI". Anthropomorphization chooses the words we use next. Thoughts, limitations, hallucinations, etc. all belong to thinkers. If we want to prove that an LLM is indeed a thinker, we must be careful to avoid these words, otherwise the narrative itself is a circular proposition: true because you said so.

> LLMs are ‘just a next-token predictor’. The only problem with this phrasing is the implication of the use of the word ‘just’.

I disagree. I think the problem word is "next". The structure of an LLM's model is not one dimensional, and the path of a continuation function is not linear. The LLM is also not alone in this story: there's also the original human-written text it was made from, and the human-written text it is prompted with. Those had to be given to the LLM.

> Let’s consider a hypothetical situation...an API that performs next-token prediction...Ultimately, the best solution would be to place humans behind the API...

Before I criticize this hypothetical, let's run with it and see where it takes us. You write a prompt, and send it to the large-language-group. The prompt gets divided into words, and each word gets assigned to a human. Each human presents a list of words they think are the most likely to come next, along with how confident they are in those predictions, and the system sorts all the responses into a continuation.

What isn't happening in that story? Objectivity. The humans aren't thinking about what your prompt means. There is no categorization. There is no logic. Most of what comes to mind when I say "they are thinking" just isn't there.

> there is no way to tell what process is behind it, similar to modern LLMs.

Except that you did just explicitly answer that question. It's humans. The thing "behind" an LLM is human-written text. That doesn't put the LLM itself in the same category as human.

> The humans behind the API possess real intelligence, and by extension, so does the API. Therefore, the intelligence implication falls apart, as something being a next-token predictor doesn’t preclude it from possessing real intelligence.

That's all very nearly accurate, but it doesn't help the overall argument. All we are saying is that real intelligence is present, and that the LLM extends that intelligence. What we absolutely failed to prove is that the LLM is intelligent or that it does intelligence.

Reason is a verb and a noun. An LLM does not do reason. An LLM does not contain reason. An LLM only presents reason.

I do.

It took me a while to understand this but YES THEY CAN. AND HOW!

The prediction of the next word in the latent space (and to a lesser extent, in images etc) is a great HARNESS for training them to develop INTERNAL STRUCTURES that start to REASON about inputs and produce OUTPUTS that actually make humans very impressed.

Think of it as a a BILLION LINES OF CODE in a non-von-neumann architecture. Instead of code in an imperative language, with some space for data etc. it is a different architecture where the BILLIONS OF WEIGHTS are the code. And it was not written by humans but discovered through a search through program space.

It would be like iterating through the space of all possible programs, to create a program that can REASON ABOUT, say, the movement of the stars and Kepler’s Laws could be just a low-dimensional projection of a much more intricate system.

It may even have elements of recursion, allowing Chomsky grammars and much more. With more training it can start to use concepts, and modify them etc.

In short — it forms DURABLE STRUCTURES in the model that lets the program REASON ABOUT inputs in the same way that people would!

But unlike people it can have access to a vast memory with perfect recall and indexing. AND on top of that, if it is programmed in a certain way it can build up concepts of arbitrary complexity and continuously select the most powerful ones (training itself, essentially) getting powers far beyond mere “reasoning”.

But now it can be done at 1000x the speed and parallelized, eg a 5 hour video can be analyzed by 100 machines in parallel and then grokked, analyzed, transcribed, clipped, and catalogued among other knowledge, within 5 seconds instead of watching it linearly. Then you can ask questions about it. But that is only the beginning. It can search across millions of videos it ingested and know exactly where every word was said, thereby for example producing a much more persuasive multimedia argument for ANY position, regardless of how ridiculous, than a team of 10 people doing it manually.

Then it can make accounts on youtube and tiktok, that seem like human ones, or at any rate attract a lot of followers, and outcompete the human accounts in both quantity of likes, followers and accounts. After all, it is just a bunch of measurable metrics to optimize 24/7, across the entire bot swarm and coordinated accounts. Then as they get enough of an audience they can literally post any info, supported by the above “reasoning” engine, and it will be more than enough to convince most people. And the ones who are skeptical will be silenced by their friends and neighbors, the way people who question eg the safety of vaccines, progressive movements, or certain wars, are.

In fact — to show the dangers of this, I could build a website that literally makes extremely persuasive videos for flat eartherism, 9/11 trutherism and convince people birds aren’t real, and much more. Then clip it and have coordinated posting of well produced clips on YouTube and TikTok, using “fair use” clips cherrypicked across millions of source videos. And then have swarms of bots on social networks (eg operating occasionally in grandfathered accounts) respond, upvote, downvote, retweet and gradually shift more people to believing it. Then can change wikipedia the way Stephen Colbert did with tripling of elephants, but with bots. To show how disinformation could spread. It must be done ethically. If someone would like to discuss how to do this experiment / warning project ethically you can contact me (see my profile).

I prefer that the dangers are highlighted early, because people are very complacent and have no idea what the bot swarms can do. Asking if they can reason is like asking if computers can finally play sounds. They can do far more, and quickly turn the internet not just into a dark forest but quickly overwhelm our society with very believable bullshit and gradually destroy the concept of truth completely.

Humans don’t walk miles when they can use cars. Now “the computer” is no longer the bicycle for the mind. It is the self-driving car LOL