Hacker News Reader

Comment by minimaxir | Hacker News Reader

Generation is more expensive than the prompt input (for Claude v1, generation is 3x the cost; for GPT-4 it's 2x the cost)

It makes the economics slightly trickier.

View on HN

I wonder why this is? Naively there's no difference between the two from a transformer standpoint.

Perhaps it's because under the hood there's additional safety analysis/candidate generate that is resource intensive?

Well each additional token generated requires rerunning the model right to find the next likely token given the previous one

Naively, yes, but you can cache the bulk of that "rerunning" [1]. That said the (non-flash) attention costs go up with the length of the sequence so perhaps this is just a simpler way to approximate these costs.

[1] https://kipp.ly/blog/transformer-inference-arithmetic/

Normally the inputs are padded out to the context length [1] and so the cost to embed 1 token or N tokens is the same. The output is produced token-by-token and so the amount of GPU time increases with the number of output tokens.

[1] I'm not sure if these huge context lengths are achieved the same way (i.e. a single input vector of length N) but given the cost is constant for input I would assume the resource usage is too.

This doesn't match my mental model (or implemented model in the case of GPT2) of how self-attention works (you need to calculate the residual stream for each individual token, attending to all prior tokens before it). Have a link?

I work on infrastructure for serving large language models but I don't have any background in ML, so my perspective is looking at these models as a black box (and also conversations with the people that do the ML stuff). It is the case in practice at least from a latency side that with a fixed context length N, embedding any number of tokens from 0 to N takes the same amount of time. Perhaps it's a difference between the conceptual and actual implementation on GPU?

edit - This occurred to me after the fact but I wonder if the difference is that the use case I work with is processing batches of many different embedding requests (but computed in one batch), therefore it has to process `min(longest embedding, N)` tokens so any individual request in theory has no difference. This would also be the case for Anthropic however.

Ah, you're thinking about embeddings which are basically the encoder stack on a traditional transformer architecture. Modern GPT-like models (including Claude), however, drop the encoder and use decoder-only architectures.

I could imagine something where encoders pad up to the context length because causal masking doesn't apply and the self attention has learned to look across the whole context-window.

Decoder only architecture? What is this? That doesn't sound like a transformer at all, are you saying gpt4 uses a totally different algorithm?

Nope, a decoder only transformer is a variant of the original architecture proposed by Google [1]. All variants of GPT that we know about (1 through 3) all roughly use this same architecture which takes only the decoder stack from the original Google paper and drops the encoder [2]

[1] Original Google Paper - https://arxiv.org/abs/1706.03762

[2] Original GPT Paper - https://s3-us-west-2.amazonaws.com/openai-assets/research-co...

How can it work without an encoder?

Everyone serious batches together short prompts so the cost is roughly proportional to the tokens.

It's because the input tokens can be batch-processed in a single forward pass through the model, while generating tokens requires one forward pass through the model per token.

If you do the math of how much memory bandwidth is required by a forward pass vs. how much compute, you'll see that inference is entirely limited by memory bandwidth and will use compute resources very inefficiently. In contrast, input processing is able to fully use the available compute.

Of course, there are ways to mitigate this problem, like processing multiple token streams in parallel, but the fundamental problem remains.

Ah, this makes total sense. I was thinking about FLOPs in the abstract and not about the wall-clock time. Thanks for the explanation.

Reply on news.ycombinator.com