Hacker News Reader

One of the notable tricks the various LLM serving frameworks provide is a special approaches to batching, e g. continuous, persistent, or in-flight batching depending on the inference framework. At some level they each allow you to start a new generation while in the middle of one or more previous generations.

Is that possible with "just" pytorch? Could it be added to gpt-fast?

View on HN

Yeah it's certainly possible, but it's not the focus of this implementation, which is more latency focused (so BS=1).

yeah i'm curious how this would stack up to vllm in a batch setting

long-term, bodes well as both methodologies should be combineable, just curious for current point-in-time wrt being relevant for production use

I wouldn't recommend using it for a batch serving setting today. One crucial optimization for batched serving (which you need if you have a large number of requests) is continual batching, which this implementation doesn't have.

Reply on news.ycombinator.com