Comment by roenxi | Hacker News Reader

Comment by roenxi | Hacker News Reader

Parent Full thread

> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)

It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.

> but at least I would still call it a "next token predictor"

We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.

Diffusion LMs denoise a canvas which I personally find more interesting.

I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.

there are some diffusion text models.

That is logically equivalent to a serial token prediction engine. If you have a diffusion text model you can use it to implement a serial token generator and if you have a serial token generator you can use that to implement a diffusion-generated text string. Don't think about the efficiencies of that, it is an upsetting idea (eg to generate N tokens, the serialising model might recalculate the same string N times from an input and emit one token of it each time - rather wasteful).

It is similar to how everything ends up being Turing complete. Any prediction system has to be equivalent to some sufficiently complicated text generation system to describe the prediction. And any text-generation system has to be equivalent to a sufficiently complicated model that serially emits tokens.

Reply on news.ycombinator.com