Hacker News Reader

Hmmm... Why wouldn't this be handled like other end of prompt things like the current mode?

Prompt caching only works based on the prefix. Let’s call your output Y and the low reasoning prompt A and a medium reasoning prompt B.

Previously you were at A+Y. Switching to medium reasoning makes it B+Y. There’s no prefix which can be cached, so the entire B+Y needs to be reprocessed.

Why can't effort just be at the end of the prompt? My understanding is that that is how modes and other "dynamic updates" like the time are handled.

Maybe switching effort routes you to a different rack of gpu’s which don’t have the cache

The KV cache is probably offloaded to RAM after a generation is complete. Then it is pulled back into any rack in the data center that has the model you’re using.

That makes sense to me. The output styles work the same way.