Hacker News Reader

Comment by dTal | Hacker News Reader

They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong.

And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.

Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.

View on HN

Balancing parentheses requires semantic understanding?

Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

It requires some level of semantic understanding, like what a paren is and what it means to balance them.

The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.

A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.

In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate, so that's fine. Please look this up if you have any doubts.

Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.

But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.

>LLMs use a neural network to interpolate

"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.

Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?

LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.

>>Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?

you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms

I guess we should declare victory. You've acknowledged how LLMs actually work.

Ah, OK. I called her a liar not because of a dispute over what exactly transformers are doing inside, but because in TFA we see:

> What are the most common misconceptions about the “stochastic parrots” metaphor?

Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”

But in the paper itself we see her say exactly that, several times. What she's trying to do now is wordsmith out of it by claiming that in her world LLMs are totally unrelated to AI, so when she said LLMs are stochastic parrots she wasn't making a claim about AI.

Nobody else defines AI to exclude LLMs, nor did they at the time, and it wasn't the core of the argument she made either. But then she admits that the paper does "generalize towards AI" at the end. So... whatever.

It's morally important to reject this stuff. When academics play word games it devalues all institutional output.

I see. Ok.

Reply on news.ycombinator.com