Hacker News Reader

Comment by mg | Hacker News Reader

Currently, LLMs start from scratch for each output token, right?

Lets say you ask an LLM

    What makes bananas yellow?

And it replies

    Bananas are yellow due to a pigment called bromelain.

I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain.

Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?

View on HN

Maybe look at it another way: ask GPT to complete the following

  Bananas are yellow due to a

  Bananas are yellow due to an

In the first case it might respond

  Bananas are yellow due to a pigment called bromelain.

In the second case it might respond

  Bananas are yellow due to an organic compound called bromelain, which is a yellow pigment.

So in either case GPT could have picked "a" or "an" without any impact on the semantic meaning of its response. In the extreme case, you could see the LLM operating according to a dumb heuristic:

  The token following "due to" is "a" with 55% probability, "an" with 45% probability.

In reality it is of course more sophisticated than this. But this dumb heuristic would explain the behavior.

And if you didn't actually include any facts about bromelain in the pretraining data, LLMs absolutely could autocomplete this with something about "an optical illusion." GPT-3 made factual mistakes like that pretty routinely, but I recall it figured out the grammatical rules of "a" and "an."

I don't think the concept actually needs to be pre-activated as you said, though I agree with faabian that this "preactivation" probably does happen in some implicit/emergent sense.

To some degree, attention is already a mechanism to make computations from previous tokens useful later. (You can think of the KV cache as a representation of the text so far and all the models thoughts on it.) And since language models are trained on sequences end-to-end, I think this is likely to happen. Multi-token prediction encourages this behavior explicitly but only for the small n token window you define.

That said, there are many works attempting to increase the compute utilization of transformer language models (early exit, mixture of depths) and novel architectures (SSMs etc.).

Thanks for highlighting the KV cache, I’ve been wondering the same thing and hadn’t come across that or didn’t remember.

Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)

Oh huh. Why not make it stateful, like re-use and compute just the “diff” when you add a new token? Assuming it’s not that easy because each token can affect attention globally.

I think I’ve read something about this but I wonder if you could abstract attention to sentence/page levels and then only recalculate the parts that are relevant.

Because attention is all you need.

I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise)

You're not wrong that you could make an LLM more stateful. There are plenty of ideas for that but it would

a) be far more compute intensive to train and run (especially train)

B)be susceptible to all of the issues that RNNs have.

C) most importantly, it would almost certainly just converge at scale with transformers. Labs run small scale, internal tests of architectures all the time and most of them basically come to this conclusion and abandon it

1. Bananas are yellow due to a biochemical process known as carotenogenesis, which involves the synthesis and accumulation of carotenoid pigments.

2. Bananas are yellow due to a specific carotenoid called beta-cryptoxanthin, which gives the fruit its characteristic yellow hue.

3. Bananas are yellow due to a gradual increase in the concentration of carotenoid pigments as the fruit ripens and chlorophyll levels decrease.

4. Bananas are yellow due to a series of enzymatic reactions that convert starch into sugars and break down the green chloroplasts, revealing the underlying yellow carotenoids.

5. Bananas are yellow due to a change in the pH levels within the fruit cells during ripening, which triggers the production of yellow carotenoid pigments.

6. Bananas are yellow due to a genetic trait inherited from their wild ancestors, which enabled the development of carotenoid pigments as a way to attract seed dispersers.

7. Bananas are yellow due to a complex interplay between various plant hormones, such as ethylene and abscisic acid, which regulate the ripening process and pigment formation.

8. Bananas are yellow due to a metabolic shift from chlorophyll synthesis to carotenoid synthesis as the fruit reaches maturity.

9. Bananas are yellow due to a natural defense mechanism that involves the production of carotenoid pigments, which protect the fruit from oxidative stress during ripening.

10. Bananas are yellow due to a evolutionary adaptation that helps the fruit stand out against the green foliage, making it more visible to potential seed dispersers.

The output of an LLM is usually randomly sampled from the top few highest probability next token/word predictions, but the model itself has no idea which word the sampler will pick. It presumably has some conceptual plan of what could follow "a", or any of it's other suggestions, but any such plan (high level prediction) is then rethought from scratch once "a" is generated.

The model not only can, but has to, change it's mind after each word generated, so this "planning ahead" is very ephemeral - more like a freestyle rapper making it up on the fly than someone thinking deeply about how best to reply and how to express it.

> 10. Bananas are yellow due to a evolutionary adaptation

Did an LLM really make this basic grammatical error?

Yes it did (free Claude Sonnet), but presumably only because it was trained on examples of us making the same mistake!

> presumably only because it was trained on examples of us making the same mistake

That's what makes it surprising.

There has to be more going on somewhere in the system. You can ask GPT4 to describe something (Vermont) and ask it to end to description with a chosen word (house). It will then usually be able to write out a descriptive paragraph that ultimately lands on your chosen word pretty seamlessly - "...embodying a serene and rural charm that culminates in the warm, welcoming feel of a cozy house."

Models do struggle with these tests, for sure, but from an analytical standpoint, a "next token predictor" should not be able to ever correctly land on the right token 100 tokens in the future.

Edit: Thinking about it, I suppose it is possible that the model can encode a "destination" in the first token. Like a pool shot that is artfully bounced off many bumpers to hit a ball, perhaps the LLM can encode a "path" to a destination token in the first token generated. Which might be even crazier as it suggests that the model is playing a meta-game with being able to precisely manipulate the individual layers of output, even though those layers are disparate from token to token.

The "next token predictor" description is a bit too literal, and anyways incomplete. A transformer has a lot of layers (e.g. 96 for GPT-2) and it's only at the input that it has pure token embeddings (ignoring positional encoding). At the output it's a built-up embedding that's decodable into a token.

One way to think of what all the intermediate layers are doing is to consider them as levels of a linguistic parse tree with the leaves (words) at the bottom and trunk ("sentence") at the top, except in the transformer the evolving embeddings at each level contain semantic as well as syntactic information. This largely hierarchical view of language was the motivation for the transformer design.

It seems we should really think of each layer of the transformer as an independent predictor, with increasingly abstract and more semantically complete information available as we ascend the transformer layers towards the output. Predict next token is only what the transformer is being trained to do at the output layer. At the inner layers (i.e. the bulk of what the transformer is doing), it will be predicting at these higher levels of representation held at those layers.

These models do struggle (although getting better) at ending on a given word rather than starting on it, and understandably so since random sampling and continual resetting after each output token (= new input sequence) means that planning ahead at level of word specificity is simply not an option. They have to continually adapt to next sampled token, and take it from there.

I'm guessing that ability to end on a chosen word is due to continued salience of that word during generation, prediction of sentence fragments using/ending with that word, and opportunistic stopping when it has been emitted and the sentence is complete. Kind of the same way you might do it yourself if you just started talking immediately without planning, while trying to end on a given word.

This post is interesting: https://clementneo.com/posts/2023/02/11/we-found-an-neuron

The output of most LLMs is stochastic. The core LLM is given token, and outputs a set of ranked tokens, with a “confidence”, to go next. Then there’s normally a filtering and search stage, where those ranked token are either feed back into the LLM to get more ranked tokens and used to for a short probability tree. I.e. if we pick the top N-ranked tokens and put them back in, each of those tokens results in a new set of N-ranked tokens.

By looking at that tree some basic filtering is done. Such as picking the branch that has the highest summed confidence, or the branch that has the fewest repeated tokens, or the fewest tokens that match with input tokens, or more often some combination of the above, plus a random choice weighed by summed confidences.

That how you can give a LLM with complete fixed weights, which is all LLM, the same input multiple times, but get different outputs.

So to answer your specific question, it can “change its mind”. Every token produced creates a new opportunity for the stochastic output filters to pick a new path through all the possible outputs.

I wonder if this means that these types of models would perform better (or worse?) for languages that do not have this sort of forward-looking grammar.

The alternative theory is that any word starting with a vowel sound is exceedingly uncommon after ’a’ in its training set, so it doesn’t need to plan ahead, just predict the distribution of the most likely next words and choose.

Which is my understanding of how they work and the dynamic at play.

I always presumed that "a pigment" is one token.

I'm a total amateur in this field though.

Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.

If you want to just see the tokens for yourself, though, just enter some text here:

https://platform.openai.com/tokenizer

No, it isn't, at least on OpenAI tokenizer.

what constitutes a token is super unintuitive.

my gut said "pig" and "ment" on this one, which happens to be right, but my gut would also say "para" and "graph" but no, "paragraph" is a single token which falls way outside the "normal" length I see of 3-4 characters

In either case, I do consistently see spaces between characters included as part of the token following the space.

" paragraph" (10 characters) is the longest token I've seen- and now I wonder what the longest token is

New winner " communication" at 14

Reply on news.ycombinator.com