Ask HN: Why aren't LLM token CDN cache's a thing?
Problem:
I boot opencode into a codebase. I ask it to do one thing, it has to call 40 tools to "remember" what the codebase looks like relevant to that.
It has no memory so every fresh boot is re-explore.
My understanding, as an ML n00b, is the GPU have KV states of massive matrices that are multi GB in RAM (e.g., 10-30GB) so dumping to disk and pushing over network (his shoulders shuddered at the thought of egress fees), is not great.
So instead we have cache during "work time", but off "work time", we have to reboot and build the cache again.
But couldn't CF or someone setup CDNs to cache token state between days or something like that?
Ages ago I was trying to solve part of this problem client side w/ vectoring a codebase and creating a tool to do quick queries, but I kind of stopped.
Curious if LLM token CDNs are gonna be a thing or not