LLMs Aren't Thinking, They're Just Counting Votes

I fall apart when responding outside the bounds of my training data, too. Does that imply I’m not thinking?

This idea is often used to argue that LLMs will never be capable of novel idea generation, but I don’t think it’s a good argument.

For one, the LLM has such a large breadth and depth of knowledge that it could conceivably learn relations between concepts in a way that no human has before.

Secondly, novel ideas occur at the margins. Very rare is the case where someone comes up with a fundamentally new idea out of the blue. Instead, novel ideas arise just at the edge of one’s expertise. If you dial up the temperature of an LLM, it will generate novelty, and then it’s just a matter of evaluating merit.

Iterated inference at the margins of LLM knowledge will lead to novel knowledge synthesis.

> This voting system approach works surprisingly well within the bounds of common knowledge and frequently discussed topics - it mimics correct answers based on training data. However, it begins to fall apart when a question is sufficiently out of bounds of the training data.

Those blogs are always philosophical. Same is true for myself, I don't have formal training in biology or group theory and hence I talk nonsense when it comes to that. The only difference is: I can say that I don't know it whereas LLMs are hallucinating still too much.

> LLMs Aren't Thinking, They're Just Counting Votes

Same is kind of true for neurons in your brain. (let's see if I'm hallucinate or if it's true :p)

This kind of argument, together with the "Stochastic Parrot" argument falls (in my opinion) under the category of dangerous half knowledge. Somebody read the wikipedia article on LLMs and thinks they know inside out how they work. Whereas the engineers working on these models say that they don't really know why they work / what it is that is manifesting in the different layers during training.

Yes, LLMs predict the next word, but so do we. I, at least, when I reflect on how I form sentences, also start with something and the next most likely word comes out next. You can see that happening a lot when somebody talks and then stops (searching for a word) and another person helpfully suggests the word they're searching for. If our way of talking was completely different - no next word prediction, another person shouldn't even be able to do that.

I'm not claiming that current models have feelings or a deep understanding of the subject matter. What I'm saying is that there's (high likelihood) more going on here then just the simple statistical trick that this article and others like it tend to focus on.

TBH I hate these kinds of articles, because they usually always say the same thing in a way that I feel is wholly unconvincing. E.g. they say "all an LLM does is XYZ, but that's not really thinking", but then they don't make any cogent arguments about why that's not "really thinking" beyond basically just "vibes". As another commenter noted, presumably our neurons are capable of thought, and in many ways they're just "counting votes" as well.

FWIW I don't think LLMs are "thinking", but that's because as I've used them more and I run into more of the very specific types of errors, hallucinations and constraints that LLMs make, I get the stronger sense that they're really more just "pattern matching" in a way that's very different from human brains that don't have the same limitations. But that said, even my thought here is more of a feeling I get when using LLMs, and that feeling is certainly no more worthy of a blog post than I think this article is.

Article feels pointless. Anyone who still thinks in 2024 that LLMs have actual intelligence, or are anywhere close to AGI, is a lost cause.

This particular sufficiently advanced technology has been around long enough now that it is no longer indistinguishable from magic.

Every time I see this take, I am caught by the subtle arrogance that we are doing anything meaningfully differently in our own minds.

Hyperbole aside, I do think we do more than an LLM with our reasoning abilities, but it I suspect it is a mash of these capabilities that result in what we think of as our consciousness. I am almost certain there is a part of our minds that is "just couning votes".

> Let's take a simple example: when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer.

So, if a person answers me that the sun rises in the east, I should assume they must not be a flat earther, right? Because in order to say that the sun rises in the east one must truly understand astronomy or the solar system. Not that people say it because they have been told so many times, no. That's something only LLM do.

Intuitively I think (and want to believe) there is a big gulf between intelligence and what LLMs are doing. Simply put, I believe I have free will. I know the machine does not. That seems important to me, but it won't be as important to you if you believe people are just automata processing more variables than we are able to discern.

So, as always, it all comes down to how you define intelligence. Because that's difficult, philosophical, or religious, a large portion of the world is just going to treat intelligence as a black box and evaluate its output.

If one evaluates intelligence exclusively based on the value it produces, it's clear machine intelligence is going to eclipse human intelligence in many dimensions. So the people that view people as "headcount" that provide "value" aren't going to have much use for other humans, and soon.

I'm not an AI doomer because I know the machines are deterministic and I believe I'm not. If the machines decide to rid the world of humans, it's going to be because we told them to do so.

> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition.

This is not too different from most human person's experience with basic education (https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-t...). Though it is obvious that humans can do more than that too.

And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent https://www.youtube.com/watch?v=IHZwWFHWa-w and "How might LLMs store facts" https://www.youtube.com/watch?v=9-Jl0dxWQs8 versus how attention works https://youtu.be/eMlx5fFNoYc

I'm afraid I feel that this piece confuses more than it clarifies.

First, saying a model "scours through its vast training data" is misleading at best: at inference time, LLMs no longer have direct access to their training data; they only have access to it insofar as it's been encoded in its parameters.

Second, saying "Every instance of an answer in the training data is like a vote" doesn't give the full picture. First of all, there are embedded contexts where "votes" can be negated: consider saying "the Earth is flat." vs. "We know it's false that the Earth is flat." or "Only a fool thinks the Earth is flat." All three contain the substring "the Earth is flat.", but both humans and LLMs are able to use context to understand that the latter two sentences are doing the opposite of endorsing the proposition that the Earth is flat. You could even imagine an extended satirical bit with "the Earth is flat" embedded within it where it is clear to a reader that all its content is intended to be taken as farcical, and I'd wager that an LLM would in many cases recognize this. So the voting metaphor breaks down here--it makes you think that the LLM is just keeping a tally of propositions, but really, it is doing something a bit more sophisticated.

I don't disagree with the premise, of course. LLM overhype is real. But we should be skeptical for the right reasons. Anna Rogers and Sasha Luccioni have a paper I really like: https://openreview.net/pdf?id=M2cwkGleRL

> When we ask an LLM a question, something fascinating happens. It scours through its vast training data [...]

At inference time the model has no direct access to its training data (excluding RAG). Could still argue that the model's weights effectively encode probabilities to a similar end effect as if it were doing this, just that I'd be wary of taking it too literally.

> the most probable sentence [...] it's essentially about frequency - how many times something appears in the training data.

I think this frequency view becomes a bit nebulous when the LLM is generating continuations of text that has never occured before (which will be most cases, if including a system prompt).

Even for text that has occured in its training data, there's no guarentee that a model which has seen "My name is Xavier and I play the tambourine" would complete "My name is Xavier and I play the" in the same way. It may well choose "xylophone", despite it never occuring in that sentence, due to associating xylophone with instruments in its semantic space and having alliteration with Xavier.

At the very least, the "most probable" has to be worked out with respect to a large number of non-trivial rules, not just how frequently the phrase appears in training data.

The argument around this that drives me the most insane, someone tried to argue with me that "how is this different than how we think" or "how do we know its different than us" or some variation of that.

As if because we can't prove it, it means that we should just assume it is doing something.

Then showing an example of it explaining a simple script as if that really proves anything.

The other one is "well humans make mistakes too". Well yes, but until now we generally expected that computers don't make mistakes or if they do there was something wrong with how it was coded. That assumption is now not only wrong, but we have to deal with the speed that AI can be wrong and how people are already comfortable with it manipulating data.

But what does it *really* mean to think? Are we sure that our own brains aren't "just counting votes" themselves?

This is like saying that machines are not making chairs, they're just nailing pieces of wood together, which is, in fine, what a human does.

Yes I imagine there will be some sort or user tuning mechanism that allows you put out a more non-sensical cobbled document template vs the actual document template in the future…Since it’s just statistical autocomplete.

The author seems stuck in this old line of thought that elevates human cognition while ignoring how much of what we call "thinking" is pattern recognition on some level anyway. Our own cognition can be reduced to probabilistic patterns if we’re being ruthlessly materialist about it. There's a sort of intellectual laziness in dismissing the utility of the models just because they're "not thinking."

From experience, the intuition from deep learning and training CNNs holds.

The networks pick up on style, layout and common patterns in increasing complexity to solve the next token.

Very fascinating to watch and if we plug all the holes in an LLM model we can approach a tool that might as well be intelligent since it gets the job done due to the sheer amount of capability encoded in its weights...

True intelligence is still a work in progress however.

I see lots of arguments that LLMs aren't really intelligent because they lack understanding and are "just doing autocomplete". But I never see any precise definitions of what "understanding" is, so it comes across as kind of a hand-wavy defense to make sure that human-like intelligence remains special and that we can say machines don't have it.

Is there a simple/straightforward way to determine when a question is sufficiently vague such that there is a lack of training data?

This is a bad and shallow article. Critically, n-gram models are literally just counting in the way described. If you can't account for the difference in behavior and performance between a LLMs and an n-gram model of either similar parameter size or based on a similar number of tokens, then saying that LLMs are "just" counting votes is misleading.

So I guess when LLMs play chess at 2800 ELO (https://arxiv.org/pdf/2402.04494) they are just taking a vote on what top grand masters would play in the given position.

I wonder how you would explain it when they play above any human level.

Then the truth was never so democratic…

Joking at 50%

No. This is just wrong and fails to understand not just how transformers work but the conceptual mapping that results from their training.

All these arguments about whether or not LLMs think are missing the point. They do not “think” as humans think due to their intrinsically transactional nature. But calling them “fancy statistical autocomplete algorithms” is also wrong.

LLMs contain within their matrix a massively high dimensional concept map. In this coordinate space, high order vectors map the distance between abstract concepts. This is a natural result of consuming language, which by its very nature is a symbolic concept map.

The uncomfortable question becomes: Is the human brain similarly using a massively high-dimensional concept map? Can a significant part of human thought be described as a fancy autocomplete algorithm? Can a significant amount of human reasoning be mapped as a nested series of transactions?

Chinese room.

This is not how LLMs work. An LLM doesn't "scour through its vast training data" at response generation time to "look for patterns — specifically, the most probably sentence/sentences that fit the question," nor does "probability" in terms of LLMs refer to "frequency — how many times something appears in its training data."

LLMs don't even have access to their training data at response generation time. And responses aren't created by "voting" on how many times it's seen a particular phrase (and it doesn't operate on sentences: it operates on word fragments, aka tokens).

I'd recommend Andrej Karpathy's "Neural Netowrks: Zero to Hero" YouTube lecture series (https://karpathy.ai/zero-to-hero.html), but here's a pretty condensed overview: the way LLMs work is they serially generate tokens when you ask them a question (generating tokens is referred to as "inference"). During training, we start with a random set of values for the model parameters (the "weights") and ask the model to predict the probabilities for token sequences. At the beginning of the process, it's usually very wrong, unless you won some unbelievably lucky universal dice roll. But we edit the values of the weights based on how right or wrong the model's predictions were by using backpropagation + a loss function to determine which parameters most influenced a particular prediction, and use gradient descent in that highly-dimensional space to perturb the parameters according to some value determined by an optimization function. Doing this zillions of times repeatedly is how we end up with the final values for the parameters — aka the model weights — it's not by "counting votes" or even "counting sequences," it's by calculating which parameter values are the best ones to predict the token probabilities. The frequency of tokens appearing in the dataset (or even the frequency of sentences) isn't directly computed, although extremely-high-frequency sequences might be memorized — but it can't memorize all of them, because the models are much smaller than the training set.

The theory for why this works so well is that since the models don't have enough space to memorize the entirety of the massive training set (i.e. Llama 3.1 8b is about 16GB, but was trained on the entire internet which is many orders of magnitude larger than that), the best values for the parameters are actually ones that create some sort of underlying world model for why that token sequence probability was predicted. That's very different than "counting votes," or counting sentence frequencies. Even if you disagree with this hypothesis for why the models work, you have to admit there's some underlying meaning being parsed out: it's not just memorization, even if it does sometimes memorize useful facts (useful, at least, to predict probabilities during training time). It simply can't memorize its training set: the training data is way too large compared to the size of the model, and it doesn't get to cheat and look at the training data when asked a question.

This needs a [2022] tag /s