Hacker News Reader

Comment by iainmerrick | Hacker News Reader

I don't see how "it's just token prediction" is relevant to that at all.

Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.

People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?

To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.

View on HN

I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process.

Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.

If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.

Hmm, what specific differences does it make in practice?

For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.

In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.

The biggest difference is I view any interaction with an llm that doesn’t produce an artifact that can be verified with skepticism.

That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out.

My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming.

This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.

> People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?

Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true.

Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.

LLMs aren’t just blindly parroting human speech, though -- they’re not trivial Markov chains any more, copying words and phrases at the surface level.

You can argue back and forth about whether LLMs are actually “conscious”, whatever that means, but it’s clear that they can be tremendously effective and useful. They can use language to get stuff done.

Reply on news.ycombinator.com