Why your local LLM feels dumber than it is

I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.

With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!

Check your numbers, folks.

All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.

Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.

Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!

It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.

I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).

If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.

I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.

I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.

Then I got fed up and asked Claude.

Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.

Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.

Then it tuned it somewhat.

Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.

In 30 minutes.

It was absolutely amazing.

Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.

But Claude got it done for me.

Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?

Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.

For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

Yes https://news.ycombinator.com/item?id=47788385

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length

2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`

3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.

4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.

5) Often meaningfully slower than llama.cpp

6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.

> (and also because VLLM is not super Windows friendly)

llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run

  llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL

You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.

You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.

We do the finetuning only on small semantic data were it helps a lot.

I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.

There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.

On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.

There is probably a better example between discrete labeling and code though.

This is something I can reuse better.