I'm begging people to write articles themselves rather than letting Claude do it for them.
I want an expert opinion, if I just wanted to ask an LLM I have my own.
How can this article not mention the KV cache even once?
I want an expert opinion, if I just wanted to ask an LLM I have my own.
How can this article not mention the KV cache even once?
Previously, it loaded up the diff, persona and prompt, and wrote those into the first message for each of 6 sub-agent reviewers. The prompt was templated with the persona name, so was slightly different for each reviewer.
My optimized version had a common first message with diff + prompt, and a script to run to atomically claim a persona. It also runs the first reviewer before the rest to warm the KV cache, and the agent doesn't launch the rest of the subagents until the first persona has been claimed (which means that the LLM is running, and therefore the KV cache is warm). The agent has a script it runs which does the waiting for it.
Agents 2-6 only run once the KV cache is warm, and because the intro message including the diff is shared, it's there in the cache. Only the persona file is different.
In my testing this brought down the cost of a review by ~half, though of course this depends on how big the diff is, how many agent are launched, and what model is used.
I thought KV cache not being as important was obvious, as a big part of the benchmrks were focused on pricing, but in retrospect I can see how I could've made that more clear.
In the end cost is what matters most and as seen from the results the KV cache miss on the cheap model doesn't really matter as much.