Hacker News Reader

but according to Little’s law, if you improve latency, you also improve throughput, right?

If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now

View on HN

I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.

It depends on whether or not latency is the constraint of throughput in your circumstances.

Reply on news.ycombinator.com