Hacker News Reader

For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s).

This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B.

I use a 64GB MacBook Pro (M4 Max).

View on HN

I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?

For qwen, it's an additional transformer layer at the very back, it ships as part of the model.

Acceptance rate is good, but MTP doesn't help in my case because of my machine's memory bandwidth constraints (M4 Max). Turn out it's better to turn MTP off.

I find mtp=3 does well with that model, only at 4 it becomes unprofitable.

Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.

Reply on news.ycombinator.com