Hacker News Reader

Comment by mulemisterX | Hacker News Reader

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

View on HN

We are running 35b-A3b with 264k context (the model's default max) using vllm and the "frog" jinja templates: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and had good luck. We are mostly running agentic workloads though, rather than coding. 27b has a slightly higher agentic job completion rate (95% vs 92%) but the 3% trade off is worth it because the A3B is sooooo much faster, and we reprocess the other jobs with a different model. Don't sleep on the froggeric templates.

Qwen: Looking at you for a new ~35B MoE! Please and thank you

I am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.

How are you finding it when it gets that high anyway? I’ve got a 64gb Mac so the context _can_ go higher. But I find around 70-80k it goes a bit weird and starts arguing with itself and doesn’t seem to know who it is vs me anymore.

You should try Glimmer MTP. Qwen3.8 27B seems to have weird memory and caching issues on oMLX

yes, with qwen3.8-27b-4bit run via rapid-mlx i can get to about 200k

Nice.. and how many toks/sec?

Around 20

Reply on news.ycombinator.com