Hacker News Reader

Comment by geerlingguy | Hacker News Reader

No, only a couple initial tests with Ollama used CPU. I ran most tests on Vulkan / iGPU, and some on ROCm (read further down the thread).

I found it difficult to install ROCm on Fedora 42 but after upgrading to Rawhide it was easy, so I re-tested everything with ROCm vs Vulkan.

Ollama, for some silly reason, doesn't support Vulkan even though I've used a fork many times to get full GPU acceleration with it on Pi, Ampere, and even this AMD system... (moral of the story just stick with llama.cpp).

View on HN

Sadly, the reason they give is subjectively terrible:

https://x.com/ollama/status/1952783981000446029

No experimental flag option, no "you can use the fork that works fine but we don't have capacity to support this" just a hard "no, we think it's unreliable". I guess they just want you to drop them and use llama.cpp.

Yeah, my conspiracy theory is Nvidia is somehow influencing the decision. If you can do Vulkan with Ollama, it opens up people to using Intel/AMD/other iGPUs and you might not be incentivized to buy an Nvidia GPU.

ROCm support is not wonderful. It's certainly worse for an end user to deal with than Vulkan, which usually 'just works'.

I agree. AMD should just go all in on vulkan I think, The ROCm compatibility list is terrible compared to...every modern device and probably some ancient gpus that can be made to work with vulkan as well.

Considering they created mantle, you would think it would be the obvious move too.

Vulkan is Mantle. Vulkan was developed out of the original Mantle API that AMD brought to Khronos. What do you mean "AMD should just go all in on Vulkan"? They've been "all in" on Vulkan from the beginning because they were one of the lead authors of the API.

Vulkan is a derivative of mantle sure, but it is quite different than what Mantle was.

There was a period in between where AMD basically EOL’d mantle and Vulkan wasn’t even in the works yet.

I would say vulkan derives from mantle, mantle development stopped some time ago.

iGPUs (and NPUs) are not very useful for LLM inference, they only help somewhat in the prompt pre-processing phase. The CPU has worse bulk compute but far better access to system memory bandwidth, so it wins in token generation where that's the main factor.

My conspiracy theory is that it would help if contributors kept the Vulkan Compute proposed support up to date with new Ollama versions; no maintainer wants to deal with out-of-date pull req's.

Hi Jeff, I'm a linux ambassador for Framework and I have one of these units. It'd be interesting if you would install ramalama in fedora and test that. I've been using that out of the box as a drop in replacement for ollama and everything was GPU accelerated out of the box. It pulls rocm from a container and just figures it out, etc. Would love to see actual numbers though.

Great work on this!

I gave it a try on my Strix Halo laptop on Arch. ROCm and everything else worked out of the box with two commands:

    uv tool install ramalama
    ramalama serve <model>

Reply on news.ycombinator.com