Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
And it did in that case make a significant difference.
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length
2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`
3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.
4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.
5) Often meaningfully slower than llama.cpp
6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.
> (and also because VLLM is not super Windows friendly)
llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run
llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL