Hacker News Reader

Comment by sharmajai | Hacker News Reader

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.

It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).

So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.

I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.

Thanks for doing and sharing the research!

View on HN

Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.

I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.

Without knowing anything about your benchmark, it might be your harness at fault here. I say this because with longer thinking you run the risk of filling up your context faster and you need a good context compaction strategy in your harness to mitigate that.

I have heard good things about Pi which supports auto-compaction, but I can't personally vouch for it since I use my own.

You're probably partly right (though I use the same context size for all models, so there still is a difference in the model/quantization itself), but I definitely see it "thinking itself into an extremely expensive and error-prone approach without backing out". A common "failure mode" of my benchmark with <Q6 is "okay, I will implement a simulator", and when I see that I think to myself "please, no, you don't need to do that at all, you just missed a tiny thing" (but if I would type that into the session, it would void the benchmark; the idea is that I wouldn't know the solution myself after all).

Especially because Qwen3.8 27B does not seem to be good enough to implement the simulator with all its intricacies and tiny subtleties correctly. So it's churning for hours with no real progress, where it just had to explore the initial problem space a tiny bit more.

And very interestingly, even with Q6, turning off reasoning entirely does not make it fall into that trap, or many others, and it more often solves the task in record time. Both because of what I just described, but also because the endless reasoning costs a lot of token, which significantly translates to time on a home rig...

I see, maybe your use case is indeed pathological for Qwen 3.8 27b. Another thing to try if you are using llama.cpp is "reasoning budget". That makes the thinking stop after the budget has been reached and inserts a custom message you can choose, so something like "you have thought for too long, now continue with the execution ..."

For what it's worth, if you haven't already, you can also let it run overnight (if you have compaction enabled) to see if it ever gets out of that hole. The reason `xhigh` is the default is because 3.8 is trying to optimize for long horizon tasks where monitoring its every single thinking misstep might not be a good use of our time.

As they say, time is money.

In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

Computer science has known the tradeoffs between memory and compute since ages ago. The same could be reflected here.

Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.

If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound. The kv cache was still bf16, so a few ops are the same between quants.

I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)

Fair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.

If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.

I observed this as well. I have problems that get solved very quickly using non-thinking mode, but in thinking mode it digs itself into giant holes, that it sometimes doesn't manage to escape from.

Like wanting to implement an entire emulator to figure out something it could easily glean otherwise (and does, in non-thinking mode).

Relatedly, someone found that on a Qwen MoE, increasing the number of experts at the deep layers reduces the number of thinking tokens. So while you mention that the LLM is capable of thinking more if it's thinking less clearly, it looks like it's also capable of thinking less if it's thinking more clearly?

This has been validated on Qwen3.6-35B-A3B and Qwen3.8-Flash-Next, but with the caveat that it was always on quants.

https://github.com/vagrillo/llama.cpp/blob/moe-expansion/ben...

Reply on news.ycombinator.com