Stochastic Parrots: Frequently Unasked Questions

The literal definition of a model is "an informative representation of an object, person, or system". I think you mean something else though, what are you trying to express exactly?

Nothing about an LLM is “just”. In what precise sense do you mean it is probabilistic?

It isn't possible to have "just probabilistic" (maybe a philosophical exception could be made for a uniform random distribution or whatever provides the little dose of randomness required to get nondeterministic results). Probabilities are always in context of a model. LLMs model language but language itself is a model of something else. My money would have been on language modelling nonsense, but that is quite clearly not the case. Turns out it models the world and so do LLMs.

The model is the thing which is learned in order to make the probabilistic prediction with low entropy.

Thanks for your explanation, I find it much more intuitive than the paper's.

In your opinion, does a Calculus solver model certain aspects of the world?

> When you train LLMs on large volumes of text that describe logically consistent facts in a million different ways, the "logic" sort of becomes part of the grammer that the model learns. That is logic becomes a higher kind of "grammer" or a enormous set of grammatical rules that it captures. But that does not mean the model can do actual logic.

This is the kind of stuff people were saying in 2023. But it’s 2026 now and LLMs aren’t just trained by reading lots of text anymore. That’s “pretraining”, and it’s still the first stage, but LLMs also have a huge amount of RLVR training where they actually do solve huge numbers of mathematical and logic puzzles and update their weights in response. They don’t just learn mathematics from reading about it now. They learn it by doing it. That is why they can now solve hard problems and probe theorems.

> that does not mean the model can do actual logic.

But they do, all the time. (Please tell me you’ve at least put a frontier LLM through its paces in the last 6 months?) If you think they can’t do logic and reasoning, can you provide examples of specific math or logic problems that you think a frontier LLM can’t do?

> Imagine that you learn the grammar of a foreign language without learning the meaning of the words. You might be able to make grammatically valid sentences. But you will still will not understand a single thing that something written in that language describes. But that will be perfectly clear to someone who actually understand the meaning of the words.

so... back to chinese room arguments?

just because amazon worker inside is just moving folders around following rules, doesn't by default mean the room as a whole can't be corresponding to "something that doesn't understand"

denying emergence as a phenomenon isn't useful when "there are plenty of higher abstraction levels in multiple fields that still capture 99% of events and are easier to model and react to" is the counterpoint

Is your justification in dismissing Fields medalists that they are impressed by funding? Not even receiving it (I assume you say this because Tao is not funded by AI for Math, but rather an advisor for it)?

Not only would it be a leap to suggest that people automatically lose their integrity by taking funds for projects they believe are useful, especially after involvement with adjacent fields, but you are suggesting merely being impressed by a fund is enough to dismiss their views?

You also have no evidence that Renaissance Philanthropies is a front for VC companies. All news coverage indicates that they seek to be an alternative for high net worth individuals engaging in philanthropy.

Many people discovering Erdos results, engaging in Olympiads etc, are doing so with publicly available models and publish the resources used in the process.

Yes, she addresses this by denying that she's made any empirical hypothesis, but in a way that's some combination of disingenuous and confused.

She also says:

> What I am trying to do... is to help people understand what these systems actually are

Can a phrase that has no empirical content aid people in understanding an empirical phenomenon?

> the astonishing willingness of so many to... turn to synthetic text... for all kinds of weighty decisions.

Why is this astonishing, if the nature of these models as "stochastic parrots" places no limitations whatosever on their empirical capabilities, reliability, etc?

> the field of linguistics is particularly relevant in this moment, as a linguist’s eye view on language technology is desperately needed to help make wise decisions about how we do and don’t use these products

Is it wise to make decisions about a product on the basis of information that has no relevance to how it is actually likely to behave?

(It may be, if one has ethical concerns with "data theft, the exploitative labor practices", etc -- but one could have such concerns about any kind of product, not just a "stochastic parrot", and linguists are not generally academia's experts on, e.g., labor practices.)

> She says explicitly it's not an empirical hypothesis. It's just a label for how they function.

Then… what’s the point of the label, if it’s not making any empirically-meaningful claims about LLMs at all? I know that LLMs involve sampling over a distribution of output logits. I’ve written code to do it. So what? I know they have statistical elements. Yet I don’t go around calling LLMs stochastic parrots, because that label implies a whole lot of claims about LLMs that I don’t think are true any longer, like that they are just regurgitating and remixing training data and can’t successfully model structured systems (like mathematics or programming).

The Hacker News guidelines say “Please don't comment on whether someone read an article. ‘Did you even read the article? It mentions that’ can be shortened to ‘The article mentions that’.”

https://news.ycombinator.com/newsguidelines.html

Maybe, but a claim about what and LLM is not is still a claim about what it can or cannot do. And specifically:

> without any reference to meaning

is vague, but I read it as actually quite a strong claim about the limitations of LLMs. I don’t think it would be possible for LLMs to do long chains of correct mathematical reasoning about novel problems that they haven’t seen before “without any reference to meaning.” That simply isn’t possible just by regurgitating and remixing random chunks of training data. Therefore I consider the stochastic parrots picture of LLMs to be wrong.

It might have been an accurate picture in 2020. It is not an accurate picture now. What is often missed in these discussions is that LLM training now looks totally different than it did a couple years ago. RLVR completely changed the game, allowing LLMs to actually do math and code well, among other things.

> "Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."

and

> "Meanwhile you have multiple Fields Medalists (Tau, Gowers) saying they’re very impressed by LLMs’ mathematical reasoning, something that the stochastic parrots thesis (if it has any empirically-predictive content at all) would predict was impossible. I doubt Tau and Gowers thought much of LLMs a few years ago either. But they changed their minds. Who do you want to listen to?"

I don't understand how these things are supposedly incompatible.

Larger models and further other refinement reduce the "haphazardness" of produced text. A big enough model with enough semantic connections between different words/phrasings/etc plus enough logical connections of how cause and effect, question and answer, works in human language can obviously stitch together novel sequences when presented with novel prompts. (The output was not limited to sequences of n words that appeared 1:1 in the training data for any n for at least three and a half years now, if not even back to when the paper was written.)

"without any reference to meaning" veers into the philosophical (see how much "intent" is brought up in the linked post today). But has anything been proven wrong about the idea that the text prediction is based on probabilistic evaluation based on a model's training data? E.g. how can you prove "reasoning" vs "stochastic simulated reasoning" here?

Perhaps a useful counterfactual (but hopelessly-expensive/possibly-infeasible) would be to see if you could program a completely irrational LLM. Would such a model be able to "reason" it's way into realizing its entire training model was based on fallacies and intentionally-misleading statements and connections, or would it produce consistent-with-its-training-but-logically-wrong rebuttals to attempts to "teach" it the truth?

I did read it. She doesn’t mention mathematics or RLVR training once, so I assume you’re referring to my point about empirical testability. Well, I think her statement that the claim “LLMs are stochastic parrots” is not an empirical claim is false, and she’s being disingenuous there with a classic motte-and-bailey fallacy. She quotes her own original paper thus:

> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. This can seem counter-intuitive given the increasingly fluent qualities of automatically generated text, but we have to account for the fact that our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether or not they do [89, 140]. The problem is, if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language (independent of the model). Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.

Do you really think that claiming the output of an LLM “has no reference to meaning” is not an empirical claim? That it doesn’t attempt to place any bounds whatsoever on what LLMs can and cannot do? LLMs can solve some very difficult mathematical problems quite well now: see the article from Gowers that was on here recently. Do you think that the output in a situation like that “has no reference to meaning?” If so, you’ll have to explain why, because I don’t understand at all.

And also create novel math proofs.

Automated theorem provers are not new, in fact they are very old. One of the most automated is ACL2, which uses the well studied waterfall method (unrelated to waterfall development).

LLMs certainly use something similar, except they understand text as input. LLMs, especially used for marketing stunts, have way more computing power available than any theorem prover ever had. They probably do random restarts if a proof fails which amounts to partially brute forcing.

Lawrence Paulson correctly complained about some of the hype that Lean/LLMs are getting.

ACL2 even uses formulaic text output that describes the proof in human language, despite being all in Common Lisp and not a mythical clanker.

They do not think and use old and well established algorithms or perhaps novel ones that were added.