Hacker News Reader

Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it.

How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?

View on HN

Regarding the MTP, if you haven't found out why yet, the `--spec-draft-p-min 0.8`. Something to play with and of course it's non-deterministic and depends on both the model itself and whatever you're actually throwing at it.

This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198

Wow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.

Reply on news.ycombinator.com