CUDA vs. ROCm: A case study
shihab-shahriar.github.io
shihab-shahriar.github.io
ROCm feels like such a half assed product that (to me at least) feels like it's been made to tick a box and look cool in corporate presentations. It's not made with the proper mindset to compete against CUDA. Lisa Su claims they're doubling down on ROCm but to me it feels like they're falling behind relative to Nvidia, not catching up.
Banding together with Intel to support SYCL would in my opinion
1. Ensure there's a lot more momentum behind a single, cross-platform, industry-standard competitor
2. Entice other industry heavyweights like MSFT, Qualcomm, ARM etc to also take the cross-platform solutions more seriously
3. Encourage heavy investment into the developer experience and tooling for the cross-platform solution
> Banding together with Intel to support SYCL would in my opinion
Except that Intel has more control over SYCL and has repeatedly hurt AMD products with anticompetitive behavior in the past. Why would AMD permit their software to be controlled be a competitor?
AMD is executing with ROCm NOW, with multiple Top500 supercomputer wins and deployment in major cloud vendors with MI300X. Yes, the software needs to improve, but the practice of throwing out software to start over is not a good strategy.
AMD has far more momentum in the data center than INTC at present.
When it comes to comparison to SYCL, HIP is much closer to the spirit of SYCL than ROCm is. Both aim to help writing a single codebase that'll run across multiple hardwares. For now though, the trajectory of SYCL appears much more promising to me than HIP. HIP is already split in two parts for CPU and GPU, which is baffling, and neither part seems to receive much love from AMD.
SYCL can get pretty much equivalent performance in Kernels to eg. CUDA. Try looking at SYCL performance papers on Arxiv. Eg. see [1].
That isn't to say that SYCL code is optimised on every platform without tweaking - you do still need to put effort into target specific optimizations to get the best performance, like you would in the CUDA or HIP.
> Why drop ROCm (used on the world's largest supercomputer?)
Some of the world's largest super-computers / HPC applications do use SYCL for AMD! The application I'm most aware of for this is GROMACS. As to why? - because having 3 version of the same code using different programming APIs is a big maintenance burden.
The fact that GROMACS is unwilling to drop CUDA support to stand fully behind SYCL is very telling.
I looked at their current compatibility chart for rocm and it's clearly just starting.
See, for example, the world's fastest super computer. That's a whole lot more than a presentation tick box.
As someone doing relatively small scale work, on gaming cards, you just aren't the target user. While these kinds of users are well represented on forums, they're a rounding error in terms of actual $.
Switching to SYCL makes no sense. They need to unseat Nvidia, and specifically CUDA. The whole point of HIP is to clone CUDA, make it easy for users to switch and piggy back on Nvidias success.
I used to work in this space (not at AMD lol), and generally I don't think the comments on HN are fair to AMD. Obviously ROCm has a long way to go (CUDA and libs are some truly amazing work), but it's going.
Until CUDA 3.0, it was similar to OpenCL, a C dialect, however afterwards it became a C, C++ dialect, with common infrastructure PTX.
PGI targeted PTX, with their C, C++, and very relevant, Fortran compilers for HPC.
PGI was acquired by NVidia, and became the main set of CUDA compilers.
Given PTX, many other languages started targeting CUDA as well, Java, .NET, Haskell, Julia, at very least.
NVidia is now invested into a Python JIT for CUDA as well.
So yeah, while C++20 is the main language in CUDA, there is also a whole ecosystem of programming languages, that the "CUDA replacements" keep ignoring.
He didn't use gaming cards. He bought the expensive $1.5k "workstation" graphics cards. What exactly is one going to use these GPUs for, other than GPGPU? Play video games? Really?
>While these kinds of users are well represented on forums, they're a rounding error in terms of actual $.
There are only three thousand developers who have ever committed to pytorch. Compared to a datacenter contract, that is a rounding error in terms of sales. Hence AMD shouldn't waste time on letting people independently work on AMD support for pytorch. They should only work on pytorch, whenever there is a big contract.
However, switching to SYCL makes no sense because Mesa is getting SYCL support. The likelihood that the mesa drivers cause a kernel panic is much lower.
What about rocm makes it fundamentally by design that much worse? And what about sycl makes it better? Rocm is already based on many standard shared OSS projects, like llvm and clang. Often the same ones SYCL stacks are based on. What about SYCL makes it so superior that it'll make up for throwing all the current work away?
If you use the driver in Linux upstream, whether you have a good time or a terrible one varies with the age of the userspace libraries you're pairing with it.
I don't know what to do about this setup. I think it's an artifact of gearing testing towards "releases" of the entire stack with specific HPC operating systems in mind.
To do this right, each component would need to be individually solid and deal with version/interface slip relative to other components, which means the cross product of testing patterns is rapidly infeasible. I'm thinking about it but don't see a solution yet.
The fact that SYCL compiles device code to SPIR-V instead of a device specific ISA like HIP for instance. The decision to do the latter, alone, is what causes ROCm to have support for a very narrow range of cards which significantly hampers adoption.
There exists a HIP implementation that also compiles to SPIR-V, chipStar. And one can in theory write a SYCL compiler that generates machine code (maybe with openSYCL's HIP/ROCm backend?)
I personally feel many of the problems with rocm are due to lack of investment and management priority - there is no "technical" solution to that as a problem. And swapping out parts of that stack will just exacerbate those problems are now they need to rebuild everything from scratch with their insufficient resources.
One is that ROCm is the entire compute stack - driver, libraries, compilers, languages and sycl is a programming language. Sycl is more like one more language that could be built on ROCm than a replacement for it.
The second is that all the entire target market wrote all their stuff in cuda and looks totally unwilling to change to anything else. That makes HIP look like a good idea and Sycl not so much. Noone (commercially significant) is likely to port their code to Sycl to benefit from Intel's GPUs.
It's worth saying that AMD bet heavily on cross platform technology. Initially opencl, but take a look at the companies behind HSA as well. I'd say that doesn't appear to have worked at all - they've got the hassle of working with external specifications, and other companies have given up on the specs, so really noone is winning there. I think Triton is a cross company thing currently being implemented.
Finally Intel specifically is moribund. Definitely don't want to be relying on them to do anything.
On the bright side, the wide array of GPU programming languages are all very similar to each other from the perspective of the compiler stack, e.g. they all run through the same LLVM back end. So if someone decides they like sycl on amdgpu enough to write it, a lot of existing code is there to help. All open source in case it's someone outside of AMD that wants to write it.
If AMD were to move away from ROCm, they would have to adopt some other API for SYCL to use as a backend. Potentially, this could be the level zero spec from UXL foundation, but that is not the same as SYCL.
However, no changing between APIs etc. will help with troublesome drivers, or issues with OS compatibility, or documentation. Those issues are orthogonal.
AMD has a really terrible reputation for dropping old and introducing new APIs on a whim. To the extent that a lot of people in the field are completely burned out on AMD. Dropping yet another API after promising that this is "the one" would completely kill any chance of ever being taken seriously in the field.
Unfortunately, it's not always easy to get an issue in front of the right people. The W6800 is an officially supported GPU and a kernel panic is a serious bug. I'd like to try to help get this issue properly triaged. Could you send me a link to your report?
https://www.semianalysis.com/p/amd-mi300-performance-faster-...
I'm looking forward to Intel v. NVidia. Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B.
Given a 7-slot motherboard, $270 * 7 = $1890 for 112GB of VRAM in one computer. That's sweet. Compute speed would be on-par with top-of-the-line NVidia workstation GPU.
Three of those are enough to run the largest open-source LLMs at around $9000.
We're just drivers + libraries + documentation away, and Intel is not bad at drivers + libraries + documentation.
With what software?
7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity.
That's at least $3000 for the base system "before" You add any GPUs.
And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...
The rest doesn't change the big picture. I spec'ed this out before, and your numbers are a good bit off if you're willing to cut corners. The cheapest way to do the rest is refurbished server, and slightly older parts. That's <<$1000.
New parts is a little bit more, but nowhere near $3k. Motherboard can be found for under $500 when a deal comes up. I found the Supermicro MBD-M12SWA-TF-O for that. PCIe risers and cooling can be done on the cheap, if you're willing to DIY. Etc. Power draw is under 3kW. A pair of 1.5kW PSUs will do that, for under $300. It's a mess (you're not getting a formal case), but it works.
Very similar systems were built during the bitcoin boom. Look some of those up, and see how they got costs down. That was all about margins.
I would like someone more enlightened to comment on PCIe issues as well.
As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power.
I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much).
That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well.
Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...
* The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase).
* I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude.
* Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference.
A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands.
I'll also mention: The scaling here isn't quite as it seems:
1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed!
2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse.
A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)
Your point stands - GPU compute on commodity hardware is very cheap. $10k per 4U quad GPU box is probably the ballpark.
Which is a crying shame because the space could really use more competition, instead of less.
That gives me a good laugh.
When I switched from FFTW to cuFFT many years ago (~2015), the transition was very smooth, the documentation was great, and all features were supported. They even put a shim "FFTW" compatible header file in so that you didn't need to rewrite your code to make it work (leaving some performance on the table).
I somewhat don't blame them: the MI300X might be miles ahead and all, but AMD are not only oblivious to the desktop market (you know, where new ideas are prototyped) but are also seemingly actively hostile[1]. NVIDIA has people doing somewhat interesting things with a 3060 (which can eventually graduate to a 4090 or even a H100), while AMD don't what to hear about it unless you have a "pro" GPU. Definitely a case of dollar-wise and penny-foolish.
[1] https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_suppo... * FWIW you can override this with an envar, but AMD aren't exactly forthcoming with that information.
Can anyone point to an example of good documentation for a big software system where they can also sketch how that was achieved? E.g. Cuda's docs are pretty good but I've no idea how they came to be, or how they stay up to date. LLVM's docs are a small amount of handwritten webpages which correlate with reality to some extent, the source for which lives in the same repo as the code.
I have an idea that it needs to combine programmers writing some things, some testing infra to notice things like internal links will 404 and some non-developers writing things.
I started trying to document one of my own systems as freeform notes under obsidian and while it kind of works at the time it diverges from reality pretty quickly, and that's without trying to have anyone else working on either the docs or the system.
So what's the proper, established answer to this?
Then you hire good technical writers to write and maintain the documentation.
I understand that AMD GPUs offer better cost efficiency for F32 & F64 FLOPs, RAM, and wattage. But however, if ROCm is such a half baked piece, shouldn't that advantage be gone? What drives AMD adoption in the HPC space then?
It's a good marketing metric, but probably contraproductive the AMDs longterm success in the field. They're spending engineering time building something they'll unlikely to be able to translate into other fields.
That’s been my experience with most large enterprise software, really. Amazing at its most critical competency, core dumps when I try something a little unusual.
More generally LLM training is weird because its a supercomputing workload being executed by Silicon Valley devs. So they want to use open source frameworks, find answers on stack overflow, use cloud providers etc. And it's new enough that random PHD's can invent stuff like Flash Attention without being employed by Nvidia.
Normal supercomputer users dont care about any of that.
So, as long as they can get a given set of applications to run above a given performance threshold, alternative vendors have a shot.
Public bidding processes are different and much more reliant solely on price. As such you can expect them to pick different choices than the private market.
This can end up picking outright duds (see Aurora)
The DoE doesn't like having single suppliers for their tech. AMD/HPE's Frontier bid was a bit of a gamble from the DoE - it wasn't remotely obvious whether they'd be able to deliver it or not, and was against a background of Intel broadly failing to deliver Aurora - whereas nvidia was a known safe bet. However placing a big order with Intel and another one with AMD was their best shot at getting away from being wholly reliant on nvidia.
Frontier shipped. It's a real thing, people run code on it. I'd guess the DoE labs talk to each other to some extent and thus AMD ended up winning the El Capitan bid. That means AMD has a flagship HPC machine that sales people can point to and a big ongoing revenue stream to continue funding development from. It looks like the DoE plan to have two HPC vendors has worked.
Aurora seems to be somewhat in existence now but is less compelling as a story other potential customers might want to copy. I'm curious whether Intel end up making a loss on it.
I really just wished my employer would give up on AMD for GPUs.
I couldn't even figure out what I needed to buy, so I didn't.
It seems that in an ideal world PyTorch support for AMD wouldn't rely on ROCm, but rather be based on high level code compiled to MLIR with AMD target support, with this same MLIR representation supporting Mojo and any other languages such as Julia that want optimized AMD support.
Developer friction is huge. Don't discount the amount of boilerplate it takes to get started with something.
So you end up having to do a lot more work to port stuff over. When porting stuff to CUDA, you can share a lot of code. Simplifies the task so much.
CUDA is C, C++ and Fortran for at least a decade, and with PTX created a third party ecosystem that SPIR still doesn't have to this day.
If you are using a single line of code to execute a CUDA kernel then you are accepting a huge amount of default behavior.
I mean, this is really, really nice and you get some pretty good performance. You're going to outgrow that single line of code very quickly. I think you're beyond it by the time you get done with Nvidia tutorial #2.
https://twitter.com/TheSixFiveMedia/status/17371772214904505...
What they need is:
1) Make one API [1] that builds out of the box, works really well and gets the best out of their hardware right away. Bonus points if its not a total pig to use.
2) Take the thing that they built for 1 and make a straightforward python binding for it that works for tensorflow and pytorch.[2]
3) Have a beer on me. Seriously I would buy them one.
Ideally there's an optional step 0 which is to stop making all kinds of press statements saying they are taking this seriously and paying sloppy journalists to say they are catching up in the race against nvidia if they can't actually get their act together to do steps 1 and 2.
[1] In c, c++ or rust so it's easy to embed in python and other languages etc
[2] By work here I mean the definition of done is a normal person can do `pip install tensorflow` and it will actually use an amd gpu with hardware accelleration right away. Not some bullshit where you have to jump through a bunch of hoops and apply a whole lot of custom patches and it kinda sorta sometimes works but actually mostly just either falls back to CPU or crashes half the time