Hacker News Reader

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.

Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

View on HN

The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.

Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.

I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.

Reply on news.ycombinator.com