Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun.

That said, after that it seems to run at the same ~35t/s speed but, I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now playing more with all the parameters like I did to get the 12B going e.g. I'm already up to ~125t/s for inputs now and with an 8bit MTP it's running about 45-85t/s output when its coding.

I run with `--spec-type draft-mtp --spec-draft-p-min 0.8 --spec-draft-n-max 8` and when it was that fast (during actual code generation) got some `draft acceptance = 0.91801 ( 3482 accepted / 3793 generated), mean len = 7.92`. I also tried 16 with no real improvement for those phases but it seemed slower, wasting too much with low 50% acceptances during other phases like thinking.

    EDIT: Oh look, I can now use larger batch sizes again and that gets prompt_eval reads back to 700+

Regarding how good it is: Probably, maybe, better than the 12B? It immediately noticed an error the 12B made, where it hadn't followed a skill's format description. But it's also annoyingly not following instructions the other one was and instead just gives me its opinion lol! And it couldn't figure out how to read a simple pi SKILL.md and run an included script. It complained that it wasn't registered as a tool. But like all other model changes, will get used to it and adjust I suppose :)