Hacker News Reader

Comment by ecshafer | Hacker News Reader

I have a self hosted Qwen 3.8 27B and I find it to be unusably bad. Using it agentically, it will spin around in circles on even small tasks talking to itself until it loses context and starts again. I even had it say "I've forgotten the users initial question"

View on HN

I have a self hosted Qwen 3.8 27B and I find it unbelievably cracked and dedicated. It's at least credibly attempted everything I've thrown at it. Just today I had it write a toy compiler with a JIT backend just to test out a concept, and that was with 4-bit quantization and 8-bit KV cache. Something has to be going wrong with your deployment.

I want a Qwen 3.8 27B hosted locally but I don't quite have the RAM for it. And, I don't want to buy the RAM until I prove I can use it.

Yesterday I did have success with Gemma-4-12b with 128k context. It fits in my RAM and it's relatively fast on my hardware.

I had to give it prompts that are quite a bit different from the way I use foundation models, but I did get it to work quite well. I feel like I could learn it's differences and get good at using it for real work.

You could try renting a GPU or VPS

I run it locally at q4_k_xl on a r9700 with kv cache bf16 and while it thinks a lot, it’s still fast enough to do the task.

This model had its knowledge replaced with reasoning ability. The chain of thought what makes this reasoning effective.

So this is why you need to let it think and don’t quantize the kv cache.

Or at least use a modern llama.cpp with KV activation rotation.

Check sampling parameters and chat template, make sure you have adequate context window, turn reasoning effort down. It should be able to one shot a small app without intervention.

Which quantization?

Reply on news.ycombinator.com