Comment by libraryofbabel | Hacker News Reader

Comment by libraryofbabel | Hacker News Reader

Parent Full thread

> The LLM, with its single-token prediction approach, follows up with a suggestion that much more closely follows the user's desires, instead of latching onto it's own previous approach.

Maybe? How would we test that one way or the other? If there’s one thing I’ve learned in the last few years, it’s that reasoning from “well LLMs are based on next-token prediction, therefore <fact about LLMs>” is a trap. The relationship between the architecture and the emergent properties of the LLM is very complex. Case in point: I think two years ago most of us would have said LLMs would never be able to do what they are able to do now (actually effective coding agents) precisely because they were trained on next token prediction. That turned out to be false, and so I don’t tend to make arguments like that anymore.

> The people behind the agents are fighting with the LLM just as much as we are

On that, we agree. No doubt anthropic has tried to fine-tune some of this stuff out, but perhaps it’s deeply linked in the network weights to other (beneficial) emergent behaviors in ways that are organically messy and can’t be easily untangled without making the model worse.

I don’t think there is any basis for GP’s hypothesis that this is related to the cursor being closer to the user’s example. The attention mechanism is position independent by default and actually has to have the token positions shoehorned in.

The human stochastic parrots (GP, not you) spouting these 2023 talking points really need to update their weights. I’m guessing this way of thinking has a stickiness because thinking of an LLM as “just a fancy markov chain” makes them feel less threatening to some people (we’re past the point where it could be good faith reasoning).

Like, I hear people say things like that (or that coding agents can only do web development, or that they can only write code from their training data), and then I look at Claude Code on my computer, currently debugging embedded code on a peripheral while also troubleshooting the app it’s connected to, and I’m struck by how clearly out of touch with reality a lot of the LLM cope is.

People need to stop obsessing over “the out of control hype” and reckon with the thing that’s sitting in front of them.

I think there’s a bit of parroting going around but LLMs are predictive and there’s a lot you can inuit a lot about how they behave just on that fact alone. Sure, calling it “token” prediction is oversimplifying things, but stating that, by their nature, LLMs are guessing at the next most likely thing in the scenario (next data structure needing to be coded up, next step in a process, next concept to cover in a paragraph, etc.) is a very useful mental model.

I would challenge the utility of this mental model as again they're not simply tracing a "most likely" path unless your sampling methods are trivially greedy. I don't know of a better way to model it, and I promise I'm not trying to be anal here

“All models are wrong, but some are useful.”

Agreed - I picked certain words to be intentionally ambiguous eg “most likely” since it provides an effective intuitive grasp of what’s going on, even if it’s more complicated than that.

Honestly, I think the best way to reason about LLM behavior is to abandon any sort of white-box mental model (where you start from things you “know” about their internal mechanisms). Treat them as a black box, observe their behavior in many situations and over a long period of time, draw conclusions from the patterns you observe and test if your conclusions have predictive weight.

Of course, if someone is predisposed to incuriosity about LLMs and refuses to use them, they won’t be able to participate in that approach. However I don’t think there’s an alternative.

This is precisely what I recommend to people starting out with LLMs: do not start with the architecture, start with their behavior - use them for a while as a black box and then circle back and learn about transformers and cross entropy loss functions and whatever. Bottom-up approaches to learning work well in other areas of computing, but not this - there is nothing in the architecture to suggest the emergent behavior that we see.

This is more or less how I came to the mental model I have that I refer to above. It helps me tremendously in knowing what to expect from every model I’ve used.

So just ignore everything you actually know until you can fool yourself into thinking fancy auto complete is totally real intelligence?

Why not apply that to computers in general and then we can all worship the magic boxes.

You’re being downvoted, perhaps because your tone is a little harsh, but you’re not wrong: people really are still making versions of the “stochastic parrots” argument. It comes up again and again, on hacker news and elsewhere. And yet a few months ago an LLM got gold on the Mathematical Olympiad. “Stochastic parrots” just isn’t a useful metaphor anymore.

I find AI hype as annoying as anyone, and LLMs do have all sorts of failure modes, some of which are related to how they are trained. But at this point they are doing things that many people (including me) would have flatly denied was possible with this architecture 3 years ago during the initial ChatGPT hype. When the facts change we need to change our opinions, and like you say, reckon anew with the thing that’s sitting in front of us.

Nah it can still be entirely on good faith.

Not everyone is an easily impressed and convinced that fancy autocomplete is going to suddenly spontaneously develop intelligence.

Reply on news.ycombinator.com