Hacker News Reader

Comment by kakugawa | Hacker News Reader

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound.

The cache is discounted for a reason. They WANT you to use it.

They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following:

1) Start charging for VRAM reservations.

2) Charge _other_ customers more.

3) Eat the cost themselves.

They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss).

Anthropic (and now OpenAI too for 5.6) prompt caching is not free.

Holding your cache in VRAM for 5 minutes or 3 hours have very different costs to them.

"They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"

So why is it 5 minutes at both peak and off peak hours? If they can guarantee 5 minutes at peak hours for everyone, why is it only 5 minutes at off peak?

I don’t think your understanding works.

If true, I suspect Anthropic will begin suspending accounts that use it.

I highly doubt that.

Keeping your cache warm is a good thing, caching saves compute and electricity.

Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.

If that was true then Anthropic would have built this functionality in themselves.

They did. On any long running query with large context, tell Claude “give me a status report every 4 minutes to keep the cache warm”. It’s entirely happy to oblige and usually says “good idea, that will keep costs down”

Reply on news.ycombinator.com