NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs | Hacker News Reader

NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs

developer.nvidia.com

69 pointsmkaushik21 comments

> Additionally, TensorRT-LLM includes fully optimized, ready-to-run versions of many LLMs widely used in production today. This includes Meta Llama 2, OpenAI GPT-2 and GPT-3, Falcon, Mosaic MPT, BLOOM, and a dozen others, all of which can be implemented with the simple-to-use TensorRT-LLM Python API.

Is this a typo? They mention it includes an optimized version of GPT-3, but GPT-3 is closed source.

The weights of gpt3 weren’t released, but the architecture is pretty close to gpt2 and explained in the paper: https://arxiv.org/pdf/2005.14165.pdf

I think the “ready to run” might be more of a byow (“bring your own weights”) kinda thing.

The code is known, nothing special. Just the weights are missing.

"NVIDIA TensorRT-LLM is now available in early access and soon will be integrated into the NVIDIA NeMo framework—part of NVIDIA AI Enterprise, an enterprise-grade AI software platform with security, stability, manageability, and support. Developers and researchers will be able to access TensorRT-LLM through the NeMo framework on NGC or through the source repository on GitHub.

Note that you must be registered in the NVIDIA Developer Program to apply for the early access release. You must also be logged in using your organization’s email address. We cannot accept applications from accounts using Gmail, Yahoo, QQ, or other personal email accounts."

Okay, kind of a bummer.

Is there anything special about H100/A100 GPUs that would make this acceleration work on them and not work on 30x0/40x0 GPUs

NVLink is the most notable (>600GB/s GPU-to-GPU communication) whereas the best case scenario on 30x0/40x0 GPUs is peer-to-peer PCIe (~30GB/s) and average case is routing PCIe through the host (~15GB/s)

3090 does 4x nvlink each way, ~100 GB/s bidirectional data

Oh huh I didn't know about that!

Normally it's architectural changes such as FP8 support as noted in the submission, but in that case newer mid-range GPUs like the A10 and the L4 (https://www.nvidia.com/en-us/data-center/l4/) should be supported but aren't.

https://github.com/NVIDIA/TensorRT/issues/982

Looks like tensorRT does work with 30x0, but I couldn't find much.

From the article I understand it's only for H100? Both the title and the charts imply that.

They just want to sell some expensive hardwares?

From the linked article: "TensorRT-LLM consists of the TensorRT deep learning compiler and includes optimized kernels, pre- and post-processing steps, and multi-GPU/multi-node communication primitives for groundbreaking performance on NVIDIA GPUs. It enables developers to experiment with new LLMs, offering peak performance and quick customization capabilities, without requiring deep knowledge of C++ or NVIDIA CUDA.

TensorRT-LLM improves ease of use and extensibility through an open-source modular Python API for defining, optimizing, and executing new architectures and enhancements as LLMs evolve, and can be customized easily."

Showing roughly 1.5-2.0x speed ups on a couple of example workloads under fp8. Not bad for what's basically just a software change. Just one more thing that will make it hard for AMD to catch up.

Projects like this are the tip of the iceberg...

While ROCm users celebrate finding just the right configuration to get PyTorch not to crash, Nvidia shops are deploying solutions in production with stuff like this, FasterTransformer, Triton Inference Server, etc, etc.

People (fairly) complain about Nvidia price points but they put a lot of work in on the software side to get a lot of value out of their products.

At the point you’re deploying with a fully tuned and optimized stack built on dozens of Nvidia software components that get 10x the perf of AMD you’re thrilled you only paid 5x as much for the hardware.

Sounds similar to vLLM: intermediate level of precision, tensor parallelism. vLLM is the most efficient server for concurrent mode. I don't know if TensorRT-LLM also implements the paged kv cache. That makes interacting repeatedly with a reference text (and long chats) very efficient. I got up to 3000 T/s prefix and 400 T/s completion on 30 concurrent requests running on 2xA6000. So it is good for batching mode or agents that run many requests.

vLLM has healthy competition. Not affiliated but try lmdeploy:

https://github.com/InternLM/lmdeploy

In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache.

If you look at the PRs, issues, etc you’ll see there are many more optimizations in the works. That said there are also PRs and issues for some of the lmdeploy tricks in vllm as well (AWQ, Triton Inference Server, etc).

I’m really excited to see where these projects go!

Does TensorRT-LLM improvements stack on top of lmdeploy or is it a replacement/competitor? Cause if it's the latter it's not so exciting since lmdeploy already advertises huge speedups.

lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons.

It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly:

https://github.com/vllm-project/vllm/issues/541

How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.

Hopefully someone cracks the software to run on cheaper gaming RTX cards.

How much is it to rent a VM in the cloud with an H100 or A100 GPU?

H100 is about $4 an hour and A100 is about $2 an hour at public market rates. But if you're willing to put up with some caveats and know where to look you can get them at roughly half of that each.

Reply on news.ycombinator.com