Hacker News Reader

Comment by ianpurton | Hacker News Reader

I've just finished writing an article on hardware required for LLM's in production https://bionic-gpt.com/blog/llm-hardware/

Would be interesting to see how this performed running Llama 2 7b.

View on HN

Last month I did testing on the latest gen AMD APU (7940HS @ 65W - 8C Zen4 CPU, Radeon 780M RDNA3 GPU) on how llama2-7b (4-bit quants) performed w/ a variety of inferencers (llama.cpp, exllama v2, mlc w/ CPU, CLBlast, ROCm, and Vulkan backends) and only llama.cpp's ROCm implementation convincingly beating CPU inferencing.

Even then, interactive (batch=1) inferencing was still barely faster. I suspect memory bandwidth (I ran w/ 2 x DDR5-5600) is the main bottleneck. Just as a point of reference, the inferencing performance is roughly in the ballpark of (but slightly lower than) my 16GB M2 MBA.

Details: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...

Yes. Without batching, every weight is only used once per token. On discrete GPUs, you generally need around ~20 flops/byte of memory bandwidth to be compute bound. I don't know where APUs are in that regard, but probably not entirely different. Memory bandwidth divided by model size is more or less the token rate you should expect to get from a good LLM inference implementation.

2 channels of DDR5-5600 = 89.6GB/s theoretical max bandwidth and a Q4_0 quant takes about 6.7GB of VRAM used, which is 13.4X, so that maps out pretty well for inferencing speed (I think the divided by parameters maybe only works for 4bit in this case). Interestingly, the ROCm implementation does go a bit faster (>15 t/s so it must have some tricks up its sleeve).

The Radeon 780M does 8.9 TFLOPS of FP32, 17.8 TFLOPS of FP16, so if 20 flops/byte applies, it has roughly 10X more compute than memory bandwidth for inferencing.

A few things:

You're a bit apostrophe happy. Sure, as a matter of style and preference you can use apostrophes to pluralize short words / acronyms, but "it's" is a contraction for "it is".

An AMD Ryzen 2700X system for $1500? That might've been a reasonable price in 2018, but you couldn't reasonably spend that kind of money nowadays if you tried.

I'd love to see some more numbers showing implementations that specifically target certain platforms. For instance, you can run LLaMa on a Raspberry Pi, if you really wanted, with certain compromises. How would a fleet of small, inexpensive systems compare with a CPU-based and GPU-based systems?

It's interesting, all the same :)

> "it's" is a contraction for "it is".

It's true, but it seems orthographically inconsistent with other possessive forms; after all, "John's" can be a contraction for "John is," or it can be the possessive form of the name "John." By analogy, one might think that "it's" can also take on both meanings, but that's not the case.

Reply on news.ycombinator.com