Comment by mike_hearn | Hacker News Reader

Comment by mike_hearn | Hacker News Reader

Parent Full thread

The authors were wrong about their core thesis and are now lying about it. That's the only criticism needed. They said, quote:

> LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form

... which is presented as unarguable fact, yet is untrue. It was obviously wrong at the time it was written and it's been proven wrong in many ways since. Worse is that they're still at it. In the article she's saying:

> Q: What are the most common misconceptions about the “stochastic parrots” metaphor? Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”

Her name is on a paper titled "On the danger of stochastic parrots". It has a section titled "Stochastic parrots" and in section 6.1 it says:

> An LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.

She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.

No, they were correct. In fact an LLM stitches together stuff it observed in its training data. That scales up way better than a lot of us expected, but it's still correct.

If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.

It's not correct. Please read some more research papers, this isn't what people working in AI believe at all. You can prove with experiments that different human languages get translated to the same abstract conceptual space in the middle layers, for example. It's why interpretability is so difficult.

The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.

They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong.

And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.

Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.

Balancing parentheses requires semantic understanding?

Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

It requires some level of semantic understanding, like what a paren is and what it means to balance them.

The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.

A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.

In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate, so that's fine. Please look this up if you have any doubts.

Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.

But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.

>LLMs use a neural network to interpolate

"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.

Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?

LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.

I guess we should declare victory. You've acknowledged how LLMs actually work.

>>Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?

you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms

Ah, OK. I called her a liar not because of a dispute over what exactly transformers are doing inside, but because in TFA we see:

> What are the most common misconceptions about the “stochastic parrots” metaphor?

Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”

But in the paper itself we see her say exactly that, several times. What she's trying to do now is wordsmith out of it by claiming that in her world LLMs are totally unrelated to AI, so when she said LLMs are stochastic parrots she wasn't making a claim about AI.

Nobody else defines AI to exclude LLMs, nor did they at the time, and it wasn't the core of the argument she made either. But then she admits that the paper does "generalize towards AI" at the end. So... whatever.

It's morally important to reject this stuff. When academics play word games it devalues all institutional output.

I see. Ok.

I read your comment multiple times and sorry to say none of your criticisms make any logical sense in any shape or form whatsoever.

>> > LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form

did you read the paper carefully??? this line is directly cited from another paper (pay attention - it's from july 2020) . the full line is - LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form [ 14 ]

the paper its cited from is titled - Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

>>>She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.

did you read the article after that line or was there a shortage of attention span??

> this line is directly cited from another paper (pay attention - it's from july 2020)

It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?

> did you read the article after that line or was there a shortage of attention span??

I did. It's more of the same, so there's nothing to say about it.

Reply on news.ycombinator.com