Hacker News Reader

How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request queuing, HTTP & GRPC interfaces, configurable delays for dynamic batching, multiple models per GPU, integrations for metrics and tracing, etc. etc.

Is Inference looking to compete directly with Triton, or in a different niche (CV)? Does it make a different set of tradeoffs?

View on HN

Inference operates at a higher layer of abstraction (tasks & architectures vs weights), targets a different audience (software engineers vs deep machine learning experts), and is focused on computer vision specifically (which informs features like its data integrations & interfaces).

I could even imagine a future version of Inference using Triton as an option for its inference engine behind the scenes.

Edit: I'd also be remiss to mention that NVIDIA have been great partners to Roboflow and we also have integrations with Tao Toolkit[1] & Omniverse[2].

[1] https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/

[2] https://developer.nvidia.com/blog/how-to-train-a-defect-dete...

Reply on news.ycombinator.com