Hacker News Reader

Comment by andy_ppp | Hacker News Reader

I hear a lot about CUDA and how bad ROCm is etc. and I’ve been trying to understand what exactly CUDA is doing that is so special; isn’t the maths for neural networks mostly multiplying large arrays/tensors together? What magic is CUDA doing that is so different for other vendors to implement? Is it just lock-in, the type of operations that are available, some kind of magical performance advantage or something else that CUDA is doing?

View on HN

1. Driver stability

2. Works on more consumer grade cards

3. Ecosystem advantage (lots of software developed against an existing and well supported ecosystem)

I have a laptop with a mobile 2060 and a desktop with a top-of-the-line consumer 7900XTX. As of yet, the 7900XTX isn't officially supported (and I haven't bothered to go down the obnoxious rabbit hole to figure out how to compute on it). Meanwhile, I can load up CUDA.jl on my laptop in mere minutes with absolutely no fuss.

Edit: if there are any GPU gurus out there who are capable of working on AMDGPU.jl to make it work on cards like the 7900XTX out of the box and writing documentation/tutorials for it... start a Patreon. I bet you could fund some significant effort getting that up and running!

As of today, there is zero consumer card support from AMD. It is an option only if you have a PRO card.

"Formal support for RDNA 3-based GPUs on Linux is planned to begin rolling out this fall, starting with the 48GB Radeon PRO W7900 and the 24GB Radeon RX 7900 XTX, with additional cards and expanded capabilities to be released over time." [0]

[0] https://community.amd.com/t5/rocm/new-rocm-5-6-release-bring...

Right, which SUCKS. Everyone who wants to prototype on their existing gear before jumping into a big pro card purchase is stuck with Nvidia, and the availability/performance of the software stack shows it.

Or even just have some hands on time to get familiar with the flow, to dick around, to build skills, to teach etc, but you CANNOT DO THAT with AMD

Exactly

I’m asking at a lower level than this, CUDA presumably has a list of functionality for GPGPU stuff like tensors, loading data, splitting up training, and building pipelines of networks /attention stuff that can efficiently fit neural networks to many sorts of data.

Why is it so difficult for other manufacturers to provide a compatible layer? If Apple can make Direct X 12 work on Apple Silicon surely AMD should be able to make CUDA (which has to be much simpler that DX12) work on their graphics cards? Is there some fundamental architectural differences that stop this from working?

There might be intellectual property concerns with "directly" implementing CUDA, and the architectures are (as I understand it) a bit different. That doesn't explain why they don't support something with similar broad compatibility though, as the actual card capabilities are very similar.

There's nothing conceptually hard but it's really a lot of work. In addition to the items you listed there's the actual compute kernels or compiler to generate those, and then porting frameworks over (PyTorch etc), and then doing the level of testing, documentation, and ongoing maintenance to make an alternative platform a reasonable idea for end users. The pitch for buying NVIDIA hardware is that existing tools, example code, and third party research will more or less work and perform well out of the box.

Edit: Going back to your original question, the main thing that makes CUDA so special is NVIDIA has already poured billions of dollars into all of this infrastructure and credibly will keep doing so.

Sure, AMD could write a CUDA emulator (if it was legal) for AMD GPUs, but if it's one tenth the performance, whats the point?

There's no real reason it would need to be 1/10 the performance though, depending on the kernel.

Compiler. It would look a lot like HIP and run at about the same performance that a cuda implementation would.

Nvidia's software is also pretty atrocious, in my opinion. The output of various tools is cryptic, updates regularly result in a totally broken system, and things often stop working for no discernible reason. Nvidia GPUs are always the most finnicky part of a system.

A modern Linux system should have uptime measured in years with minimal effort. A modern Linux system with Nvidia GPUs will have uptime of weeks with a lot of fuss.

(I'm no expert, just someone who's managed a number of PCs and a few servers.)

Right, but they can get away with that because they have essentially no competition.

With that said, Pop!OS does a really nice job of handling the Nvidia software stack - I've been running it on the laptop mentioned above for several years with no issues (though I don't leave my machines on 24/7).

Everyone else does the work to make sure it runs on cudnn, because they bought the hardware when it was the only reasonable solution, and if it works on anything else that’s just a happy accident. So you’ll spend weeks of your incredibly expensive engineering or researcher time fighting compatibility issues because you saved $1k by going with an amd card. Your researchers/engineers conclude it’s the only reasonable solution for now and build on nvidia.

It’s classic first mover advantage (plus just a better product / more resourcing to make it a better product honestly). I think you have to be a really massive scale to make the cost per card worth the cost per engineer math work out, unless AMD significantly closes the compatibility gap. But AMD’s job here is to fill a leaky bucket, because new CUDA code is being written every day, and they don’t seem serious about it.

Yup, filling the bucket could be worth hundreds of billions though. Maybe even trillions, seems like a sensible punt.

> it’s the maths for neural networks largely multiplying large arrays/tensors together?

Yes, it's multiplying and adding matrices. That and mapping some simple function over an array.

Neural networks are only that.

It’s the inter GPU communication. Scatter and Gather have much worse performance on AMD GPUs.

Reply on news.ycombinator.com