Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by alberth | Hacker News Reader

Parent Full thread

alberth·

Is I/O starvation the bottleneck with GPUs?

I didn't think it was.

View on HN

Memory bandwidth is the bottleneck for LLM inference. That's my understanding at least.

That's correct, but also compute to some degree. The larger the model the more of a bottleneck memory becomes.

There are some older HBM cards with very high bandwidth like the Radeon Pro VII which has 1TB/s of bandwidth like the RTX 3090 and 4090, but is notably slower at inference for smaller models since has less compute in comparison. At least I think that was the consensus of some benchmarks people ran.

Isn't it only the case when inference isn't batched?

Even in a local setting, batched inference is useful to be able to run more "complex" workflows (with multiple, parallel LLM calls for a single interaction).

There is very little reason to optimize for just single stream inference at the expense of your batch inference performance.

With a typical transformer and a GPU the batch size that saturates the compute is at least hundreds. Otherwise (including typical size of 1 for local inference) you're memory bound.

GPUs have much more memory bandwidth than CPUs. Meanwhile, the ALU:bandwidth ratio of both GPUs and CPUs has been growing exponentially since the 90s at least. So, the FLOPs per byte required to not be starved on memory is really large at this point. We’re at a point that optimization is 90% about SRAM utilization and you worry about the math maybe at the last step.

For inference, it often is. Though for most consumer parts the bigger concern is not having enough VRAM rather than the VRAM not being fast enough. Copying from system RAM to VRAM is far slower.

Reply on news.ycombinator.com