Hacker News Reader

Half OT:

Why do the cache hit rates seem to vary so much between harnesses?

I use pi, which is very minimalist, and I get a hit rate of ~99%. Paying like $1 a day for Flash. Yet, the hit rate mentioned on OpenRouter is only ~79%.

View on HN

Yes hit rate does vary by harness and by how you use the harness. If you use subagents, for instance, they will start with a whole new context created by the main agent, and this will not be cached. If you mostly use the main agent with Pi, you’ll have high hit rates and low costs. Sometimes agents do “cache busting” things where they’ll move around some of the text in the context to try to keep old instructions from being forgotten, thus keeping the agent on task, and this will bust the cache. I’ve heard, but not validated myself, that Open Code has some issues with this.

BTW, this is one of the things that I really like about Pi. It’s very simple and thus very predictable.

Some harness (cc) injects system messages in the middle, this kills the cache as the model provider moves the system to the top and concatenate them together. Use homemade proxy, you will get same 99% hot rate with even cc.

Reply on news.ycombinator.com