Hacker News Reader

It's a software issue.

How so? I've had no issues running llama.cpp with vulkan compute on my AMD graphics card (9070 XT). llama.cpp exposes an OpenAI API endpoint, which seems to be the lingua franca for AI applications. Does llama.cpp not work with datacenter AMD graphics cards?

Yes it works great with them through Vulkan. ROCm is more hit and miss, especially because the cards which are affordable (and PCIe, the new ones aren't PCIe compatible!) are already fairly old and have already been dropped by ROCm.

But Vulkan is actually faster too so there is no reason to bother with ROCm anyway.

I don't currently have an actual datacenter card, but I have a Radeon VII Pro, which is basically a 16GB version of the MI50 Instinct with its own fan.

Because OP is running it on an M3 Ultra with Ollama. He'll get much better perf with llama.cpp or MLX, both of which Ollama wraps, albeit very poorly.

Reply on news.ycombinator.com