Hacker News Reader

Comment by wolvoleo | Hacker News Reader

These numbers are a lot lower than I expected from such pricey hardware.

I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.

View on HN

I am also running some old AMD datacenter cards, 2x MI25 in my case. Getting around 30 tokens/second with short context.

Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins.

ROCm nightlies do actually support these old cards, just not the ROCm stable releases.

any numbers to share? also, what inference engine do you use and with which api?

I use Llama.cpp and OpenAI-style API of course (who doesn't, unless you use Ollama perhaps?)

Ollama sucks, they dropped ROCm support for my card out of the blue, and the Vulkan replacement wasn't ready. The developer just shrugged and didn't care about the people affected so I moved to llama-server. Also, here's some other reasons to avoid ollama: https://sleepingrobots.com/dreams/stop-using-ollama/

And really, ollama are trying to sell their cloud platform and are always running behind llama.cpp in new features like the KV Cache quantisation.

So numbers, I don't really measure it from day to day, I'm not really interested in benchmarks at all. But I just did a quick test, llama 3.1 8b q8_0 does about 70tk/s (with Vulkan up from 45 when I was still running ROCm), and qwen 3.5 9b q8_0 does about 45, I think that includes its time spent thinking, not sure. Overall I'm happy with that.

Smaller sizes and quants are of course faster than this.

It's not terribly much faster than a Mac Studio, but it's one hell of a lot cheaper of course. And for the price of a Mac Studio you can get a lot faster hardware. The only thing where the studio excels is if you need large amounts of vram like 64GB and up and run the models to go with that (I run multiple small ones and spread them over multiple cards)

It's a software issue.

How so? I've had no issues running llama.cpp with vulkan compute on my AMD graphics card (9070 XT). llama.cpp exposes an OpenAI API endpoint, which seems to be the lingua franca for AI applications. Does llama.cpp not work with datacenter AMD graphics cards?

Yes it works great with them through Vulkan. ROCm is more hit and miss, especially because the cards which are affordable (and PCIe, the new ones aren't PCIe compatible!) are already fairly old and have already been dropped by ROCm.

But Vulkan is actually faster too so there is no reason to bother with ROCm anyway.

I don't currently have an actual datacenter card, but I have a Radeon VII Pro, which is basically a 16GB version of the MI50 Instinct with its own fan.

Because OP is running it on an M3 Ultra with Ollama. He'll get much better perf with llama.cpp or MLX, both of which Ollama wraps, albeit very poorly.

Reply on news.ycombinator.com