I find mtp=3 does well with that model, only at 4 it becomes unprofitable.
Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.
Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.
No comments yet.