Hacker News Reader

Comment by hypfer | Hacker News Reader

Old news by now, but you might not be aware that llama-server can do multi-model for a while now,

Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)

Then, any api client can just select a model and the system does the right thing.

It's great software. It just works.

You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.

Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.

View on HN

Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.

There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way.

https://github.com/mostlygeek/llama-swap

If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.

Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).

The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.

Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.

I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.

Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.

llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.

I use it on the daily. Haven't exactly missed ollama.

The router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously.

The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.

If you revisit my comment and pay attention to the opener:

> but you might not be aware that llama-server can do multi-model for a while now

you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing.

So the answer to the question has already been encoded as text available.

I can see the desire for explicit validation though. For that, I would propose a sentence structure like

> Oh cool! That means that llama-swap is now superseded/no longer needed?

That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare. Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.

btw, llama-swap provides a nice UI for monitoring performance and logs, and even the ability to stop an infinite session that consumes GPU resources (sometimes that happens).

Does the llama.cpp UI provide the same? If not, it is too early to say that llama-swap is “superseded/no longer needed.”

It's also convenient if you want to run separate builds behind a single config, sometimes there's a PR I want to run before it's merged

Seconding this. I’ve found that llama-cpp doesn’t completely evict all of the GPU memory with its idle sleep parameter configured.

> any api client can just select a model and the system does the right thing

My machine can only load one model at a time. The loading/unloading times just don't seem worth the switch. I tend to use qwen3.6 for anything and that's it. Then again, I am a simple coder.

> (Optimized by you through testing. Not that AI)

Why not optimized by AI through testing ? Give it a test set to work on and let it loose.

AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.

^ This.

Intent is the answer and AI has none.

This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times

What model has to trade between those? I have both. You just have different, independently-optimized forward passes for each.

I'm running on Strix Halo so memory bandwidth is my constraint. In that example I'm describing the choice between using ROCm or Vulkan. I have a llama-swap config that can call different instances of llama-server running a toolbox with either runtime.

Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.

Reply on news.ycombinator.com