Pinkert

That's actually a great question. and the answer is yes and no; While it does disable the caching mechanism for the conversation history (and not for the system prompt, who remains constant), there is a difference between a chatbot with a constant chat history (just exchange of messages) and an agent who uses a large part of the conversation as a type of "scratchpad", sometimes even holding variables value in the beginning of the chat (to be sort of 'stateful'). if these variables change, the scratchpad changes (can be even 30%-40% of the entire conversation), there is a timeout in the cache (Claude gives you 5 minutes of cache for normal caching) or any other change to the exact history - you get a recaching of the entire conversation. additionally, caching still costs money.

The main advantage of the librarian is that is an 'insurance policy' for this caching mechanism. combining it with solving the context rot issue - and you get improved performance at scale.

Thanks! I'd love to hear how you implemented it, and if you can suggest any improvements for my solution. feel free to submit PRs as well!

One architectural tradeoff we are actively working on right now is the latency of the "Select" step for shorter conversations.

Currently, the open-source version of Librarian uses a general-purpose model to read the summary index and route the relevant messages. It works great for accuracy and drastically cuts token costs, but it does introduce a latency penalty for shorter conversations because it requires an initial LLM inference step before your actual agent can respond.

To solve this, we are currently training a heavily quantized, fine-tuned model specifically optimized only for this context-selection task. The goal is to push the selection latency below 1 second so the entire pipeline feels completely transparent. (We have a waitlist up for this hosted version on the site).

If anyone here has experience fine-tuning smaller models (like Llama 3 or Mistral) strictly for high-speed classification/routing over context indexes, I'd love to hear what pitfalls we should watch out for.

Using a v3 onion address as both the cryptographic identity and the NAT traversal layer is such a clean architectural choice. No STUN/TURN servers, no hole punching, you just boot the script and Tor handles routing.

For those who use Tor regularly for things other than web browsing: how bad is the real-world latency for pushing a ~20KB Opus audio chunk over Tor these days? Are we talking a 2-3 second delay, or is it much worse?