Hacker News Reader

Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference.

Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.

View on HN

Good news, MTP support is already included in this release. Not sure why they haven't made this clearer.

I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.

It's included in the model gguf itself.

Maybe I'm not seeing the speed improvements because it's a dense model, and I'm comparing it to the 3.6 MoE with MTP. Likely, I'm attributing the speed I see to something else.

Honestly, if MTP is available, I don't know why it's so much slower here.

Reply on news.ycombinator.com