Without understanding most of that paper, here's a question for someone who might know more: can pagedattention work to make cpu inference faster too?
LLMs can't batch token generation for single users. Its sequential, each token depends on the next. In fact that's a part of the paper: "dumb" batching will leave the GPU underutilized because responses aren't all the same length, and they end up processing one token at a time at the end.
This basically enables KV cache reuse when there is a prefix matching (from my shallow understanding of how KV cache works).
I failed to see how this help for local deployed LLM, unless you consider the case you ask the same question or with the same prefix are high (like always starts with "please help me ..."?)