Hacker News Reader

Comment by Carrok | Hacker News Reader

As someone who spent the better part of a year trying to get various Nvidia inference products to work _at all_ even with a direct line to their developers, I will simply say "beware".

View on HN

Just curious what your issues with Triton were. We've done OK with it using it to serve LLM models w/ a classifier head via HF Transformers pipeline & Flash Attention 2, as well as serving text generation models with the vLLM back-end.

triton is not that bad, TensorRT will give you nightmares

100% - probably why vLLM is now the default back-end in Dynamo.

Triton is not that bad at all, considering the wide scope of systems it has to support (tensorrt, onnx, multiple generations of pytorch, cuda, python). It was much nicer than the old Torchserve project which was JVM based.

I've done very little with Nvidia software, but what I have done puts me off ever doing it again. I quit a job partially because it involved trying to get their shit to work. (There were other factors, but that was definitely on the 'GTFO' side)

Can you share some of your wisdom on setting up a scalable inference infrastructure?

Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html

As someone who has run LLMs in production, using Ray is probably the worst idea. It's not optimized for language models, and is extremely slow. There's no KV-caching, model parallelism, and other basic table stakes features that are offered by Dynamo or other open source inference frameworks. Useful only if you have <1 QPS.

Use SGLang, vLLM, or text-generation-inference instead.

This is probably true, but unlike every Nvidia product we tried, it did, you know, reply to inference requests with actual output. That said, you can serve vLLM with Ray Serve. https://docs.ray.io/en/latest/serve/tutorials/vllm-example.h...

Ray doesn't offer anything if you use vLLM on top of Ray Serve though.

It does if you need pipeline parallelism across multiple nodes.

It really depends on the task. If you have 1 massive job, Ray sucks and doesn't provide table stakes. If you have 50M tiny jobs, Ray and kuberay is great and serves as the backbone of several billion dollar products.

Good for the goose, good for the gander...

> If you have 1 massive job, Ray sucks and doesn't provide table stakes.

Can you say more?

is this in reference to Triton?

And NIM, yes.

Reply on news.ycombinator.com