Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1M?
You can put all the other source files in an initial system message and the current file in the user message. Then, if you call multiple autocompletes within 5 minutes of each other, you pay a drastically reduced price for including all of the other files in the context. Also, the latency is much reduced!
Yes, you could probably get a similar outcome by incorporating RAG, search tools etc to the autocomplete, but as a simple approach with fewer moving parts, the caching will reduce costs for this setup.
Does your approach work for you?
In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time.
Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.
I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation).
Simple queries are where I expect there to be the least gain for this kind of thing.
tl;dr can cache system prompts, tools, and user messages (up to 4 total) with better returns on massive inputs such as documents.
The use case is more for client-facing applications that would hit the cache frequently rather than internal copilots.