Continuous batching to increase LLM inference throughput and reduce p50 latency | Hacker News Reader

Continuous batching to increase LLM inference throughput and reduce p50 latency

anyscale.com

110 pointsmichellezzz20 comments

As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any!

[1] https://github.com/huggingface/text-generation-inference/iss...

As always the LLM hamster wheel is moving too fast. I’d love to matrix download half this stuff (I still enjoy learning so classic-learn the other half)

I was a bit surprised by this:

> For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"]

From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?

I suspect they’re simplifying things

The tokens are chosen by a method called BPE, and there are single letter tokens. IOW you can encode the same text many ways. That said, this is probably just shown like this for illustrative purposes.

It's mentioned in the article: "This example simplifies things a little bit because in actuality tokens do not map 1:1 to ASCII characters (a popular token encoding technique is Byte-Pair Encoding which is beyond the scope of this blog post), but the iterative nature of generation is the same regardless of how you tokenize your sequences."

It's an extremization that still is true for character-based models.

You need to use a decoder to convert the response to a sentence. Example: https://huggingface.co/docs/tokenizers/api/decoders

A number of submissions lately have "simplified" and presented every character as a token. This has only confused many readers.

LLMs use a vocabulary of statistically chosen tokens. GPT 3 vocab, for instance, splits Sacramento into three tokens-

Sac - 38318

rament - 15141

o - 78

There is a rule of thumb that about every four letters in English text becomes a token but that's just the average.

California is a single token (25284). As is Canada (17940). And so on.

Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.

The underlying reason is closer to your second observation. Statistically chosen tokens are most likely to be reusable and composable. Much less to do with hardware architecture. Token vocabulary happens before any quantization where processor word size may play a role.

It's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you can define how big you want your vocabulary to be (the number of discrete tokens), and it will find the best solution of subwords/characters in a sample set.

In the case of GPT-3 the vocabulary has a size of 50,257 tokens. GPT-4 increases that past 100k (see https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...).

It's very similar to compression algos, really. Find recurring sets of characters.

If you achieve 23x throughput without reducing the amount of work being done, does that mean GPU utilization was ~4% before?

Kinda hard to believe, but I have no intuition about this.

It's very hard to filter blatant self-promotion and marketing jargon from actual innovation (which goes at a much slower pace) these days.

The speed people are just creating wrappers or minor changes and using words like "disruptive", "game-changing", "democratising <something>" just feel so inflated and boring at this point.

I hope this gets very soon into the next phase of the hype cycle[1], so the we can talk about something else. [1] https://en.wikipedia.org/wiki/Gartner_hype_cycle

"The speed people are just creating wrappers or minor changes"

To be fair, the overhead of running LLMs at scale -- the cost per interaction -- is the limiting factor for commercial deployments. Efficiency is an enormous advantage.

So what? TGI also supports this.

1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.

Reply on news.ycombinator.com