is that impressive? I was thinking 100 tok/s on an H100 is really slow considering LMDeploy claims 2000+ on an A100 and a large batch size.
Using an H100 for inference, especially without batching, sounds awfully expensive. Is cost much of a concern for you right now?
100 tokens/s on the user's end, on a host that is batching requests, is very impressive.
With TensorRT-LLM + in-flight batching you can oversubscribe that one batch slot, by beginning to process request N+1 while finishing request N, which can help a lot at scale.
I suppose one could get decent utilization by prompt processing one user while generating tokens for another.
ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100:
https://github.com/turboderp/exllamav2#performance
I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my generation speed on a 3090.
100 tokens/s batched throughput (for each individual user) is much harder.