Ah, that's fair, and faster than any of the LMDeploy stats for batch size 1; nice work!
Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?
Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?
100 tokens/s on the user's end, on a host that is batching requests, is very impressive.
With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.
I suppose one could get decent utilization by prompt processing one user while generating tokens for another.