Hacker News Reader

Comment by m0zg | Hacker News Reader

TensorRT offers significant advantages wrt inference and it takes ONNX files. Best I can tell this does not have a TensorRT backend (https://github.com/uber/neuropod/search?q=nvinfer.h&unscoped...). Why not?

View on HN

Adding backends for TensorRT, ONNX, JAX, etc are on our TODO list (and we'd love to see PRs to add support for these and others)!

We actually do use TensorRT with several of our models, but our approach is generally to do all TRT related processing before the Neuropod export step. For example, we might do something like

    TF model -> TF-TRT optimization -> Neuropod export

    PyTorch model
    -> (convert subset of model to a torchscript engine)
    -> PyTorch model + custom op to run TRT engine
    -> TorchScript model + custom op to run TRT engine
    -> Neuropod export

Since Neuropod wraps the underlying model (including custom ops), this approach works well for us.

I wrote our internal lightweight version of neuropod at another SDC startup where we did use TensorRT. Our ML researchers worked in pytorch and more often than not, the pytorch -> onnx -> tensorrt conversion did not work. We ended up needing to replicate the network architecture using the tensorrt library and manually convert the weights from pytorch. Then we'd use the tensorrt serialization to compile the models so they could be run in c++. I imagine that they may have tried this in neuropod and saw the same conversion problems. TensorRT was a big investment to get running smoothly but it did shave off 20% or so off our inference latency

It's gotten better in TensorRT7. I'm using it quite successfully. It does have a lot of corner cases though, that much is true, and the documentation is really poor, which, coupled with it being mostly closed source, severely limits adoption.

That said, I'm getting ridiculously good performance with it, even without using the TensorCores.

Reply on news.ycombinator.com