Hacker News Reader

Comment by smcleod | Hacker News Reader

The smarter 27B is so fast with MTP I've found I really don't need the 35B-A3B. You get around 70tk/s on a M5 Max lowering to around 40tk/s at higher context sizes.

View on HN

I've benched 3.8 27B being significantly slower and less quality than 3.6 35B-A4B (both 4-bit quant, MTP, both using turboquant 4-bit served by oMLX), to the point that I'm not even using it right now (on an M3 Max). What's your use case and what did you observe? I might be missing something.

I believe you mean 35B-A3B, there was no such thing as A4B. I use 27B and other models for software development, and quite a few research or similar agents. I cannot imagine a world where the old 35B-A3B model is smarter / more capable than 3.8 27B - the difference is night and day for coding at least. Where 35B-A3B was fast and felt like a Haiku model, 27B feels like a strong Sonnet when given the right tools. I don't use turbo quant so can't comment on that, but with the A3B model you're using you probably won't get much from using MTP with small MoE models like that.

People over-quantize things, muck with the temperature and other settings based on superstitions or results from models they think are similar. There's lots of ways to make 3.8 27B dumber.

I've had great success with allowing the model to review optimal settings and my system specs. It comes up with the right configuration. Running Pi harness. I just had Ornith 1.5 take a moment to configure itself, now it's reviewing a large project I'm working on, so far its really impressive for my needs. Qwen3.8 27b Unsloth(Dynamic 3.0) is also perfect. These two are working together, and I'm in a sweet-spot, I now have all I need.

Yes, A3B, sorry for the typo. I’ve benched it and used it in practice for agentic coding, and I haven’t seen any benefit to it yet. Lightning MTP helps a lot in the perf department.

I would suggest careful benchmarking. I actually tested and benchmarked, and the new Qwen3.8-27B model is actually slower with MTP on my M4 Max. MTP only gains anything when generating long code sequences, which is very unlikely as the model spends most of its time thinking, not generating code, even if you use it for coding (which I don't).

I get 20 tokens/s on an M4 Max (larger GPU).

That shouldn't be the case, it sounds like you've got something else going on with your setup. Here's my benchmarks: https://omlx.ai/my/fadc2127d384283f5df1fcc2c093a9f95700c6a52... which are inline with the communities: https://omlx.ai/benchmarks/performance?sort=tg_tps&order=des...

I only get ~4 tok/sec on a M1 Pro with MTP.

I get ~10 tok/sec on a M1 Ultra, but the real issue is abysmal prompt processing which makes it unusable. On the M1 and M2 series MTP is actively harmful for performance.

Does 27b mean it fits one 32GB GPU?

It doesn't mean that, but yes it would (with a 5 bit quant).

27B means it has 27 billion parameters, but how that translates to model size depends on the model architecture. For Qwen 3.5 architecture (same as 3.8) the native size is around 55 GB memory. So you need quantization to fit it in 32 GB.

Go here: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

See the right hand side panel, you see a whole palette of quantizations and their respective sizes. Should give you an idea. Note that these are not the only quantizations available.

What quant, what runtime?

AWQ 5bit, oQ5. oMLX.

Reply on news.ycombinator.com