My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workload.
By caching them they resume from where it left off from before thereby completely bypassing all that computation.
For large contexts this could save a ton of compute!
I think this feature and structured outputs are some of the biggest inventions in LLMs this year.
This means that every attention layer can use previously calculated outputs for the same prompt prefix. So it only needs to calculate from scratch starting from the first unique token in the prompt sequence.
Shouldn’t be a huge deal to adjust imo
One of the bigger problems is that closed model providers don’t want to expose the embedding space and let’s users see what they have
The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.
Plus, I'm reasonably certain they are caching the attention scores anyway.