Hacker News Reader

I more relevant benchmark would be against TensorRT, the current kind of model serving performance on NVIDIA GPUs.

In the paper (https://dl.acm.org/doi/10.1145/3575693.3575702) there is also a TensorRT comparison. There, they are pretty much neck and neck. Sometimes faster, sometimes slower.

+1. There are multiple comparisons in the paper, and we plan to provide a few more soon. The typical story that if the model is at least a bit unusual, not overly optimized manually, Hidet can provide some speedup even over TensorRT.

Reply on news.ycombinator.com