Optimizing Matrix Multiplication on RDNA3
seb-v.github.io
seb-v.github.io
this is overblown at least wrt forward compatibility - all of the instructions used are in RDNA4 and most of them are even in CDNA3 (CDNA4 isn't public yet?) and the ones that aren't exactly there are only slightly renamed (ds_load -> ds_read). Sure it's annoying but it's not the end of the world to have some `#ifdef`s in your code (that's not very much different from what the compiler itself is going to do anyway).
I think this is what deepseek had done to get their speedups on older hardware.
Even way back in the days of GPU crypto mining - custom kernels hand built (mostly just unrolling loops) would yield 20% improvements over just running opencl and letting the drivers compile it down.
i swear it's so funny when people talk about this stuff like it's all weird/surprising. y'all realize that there are hundreds (thousands?) of engineers across FAANG whose full time job is optimizing CUDA/ROCm/whatever code for their team/org/company's specific workloads? like do y'all think that serious shops really just go with whatever the vendor gives you? ie none of this is in the least surprising - it's completely expected that whatever the vendor designs generically for the entire market segment will fail to achieve peak perf for your use case.
When Carmack left Meta I believe he claimed they were only getting around 20% utilization on their even then enormous GPU fleet. So I could see them also leaving a lot of perf headroom on the table.
Working on a platform that hides so many low-level details is a challenge, and the fact people have to go to such length to get access to it is noteworthy. 'maxas' was noteworthy and unneeded on many (most ?) other platforms.
Not saying Intelstuff or armstuff is 'easier' but at least you get access and are tooled to work on the actual low-level asm.
I'll repeat myself: no it's not. There's nothing noteworthy about it at all. In fact I literally cannot fathom why anyone ever expects or expected otherwise. Is it because the oft-repeated notion of "abstraction"? I guess I must be the sole programmer that has always known/understood, even from the first intro class, that abstractions are just assumptions and when those assumptions don't hold I will need to remove the abstraction.
Well done!
I wonder if there's untapped potential in a GPU language which made all of those implicit classes explicit in code, now that we've sort of stabilized on them. It wouldn't allow you to do anything that you can't already do with clever optimizations and a profiler, but it could have the potential to make the optimizations clearer.
In general I'm very curious as to why we don't have any new languages that are better aligned with current hardware. For some reason we collectively decided that it was more fun to make everything general, which is especially unfortunate considering the real world got increasingly homogeneous. Compiling to some intermediate language makes no sense when you're only ever going to run on x86 anyway.
It's probably not surprising that you can grind a decent improvement over a general solution, and many of the improvements shown here will need to be re-balanced, or even simply not work, for kernels working on different matrix layouts. Similarly for trying to work on different hardware - even in the same architecture and generation these sort of details are often changing.
And all that required going down to the ISA level, which is a lot less easy (certainly less documented) for Nvidia - for example the "inspiration" post linked [0] on CUDA didn't beat cuBLAS also didn't try modifying the SASS directly, so there might be similar level gains unrealized there.
but you're acting like they pick these numbers using a random number generator for each generation when it's just reasonable/rational stuff like "here's 2x more LDS or more registers for free because the new process node is 2x smaller". like you must realize that they're not throwing everything away and starting completely from scratch for every new gen right? incidentally, while LDS will grow and # of registers will grow, there's absolutely no way they'd change the banking - e.g., CUDA hasn't changed it since 2.0.
I'm not trying to minimize the work here, it's interesting and a good example of the sort of lengths you can go to in order to squeeze that last little bit of performance out (and again, showing the advantages of public ISA documentation and support for users working at that level), I just took issue to the parent comment seeming to use this work as evidence of a poor baseline.
The problem with ROCm isn't that it only half-utilises the hardware, the problem was that someone trying to write this blog post in 2020 would have had (or at least the probability was rather high) a heading somewhere around implementing Kernel 0 talking about how the software crashed or the kernel panicked when they tried to run the benchmarks. That was what happened to me when I tried a conceptually similar exercise. I was wandering around HN posting comments about how there were no articles like this one to be found for AMD hardware and musing whether it was technically possible to do.
This makes me wish I'd bought an RDNA3 card instead of a Nvidia one for my last purchase. Not that I really regret the choice, AMD are going to have to show that they're interested in supporting consumer cards for a little longer to get me to trust them again although they're on the right path.
If AMD was only 1 hardware generation behind Nvidia they'd be pretty competitive. People are happy using CPUs with a gap of several generations from the cutting edge. And it isn't even that bad because anyone who particularly cares can optimise their software and avoid using rocBLAS.
And while a possibility may exist to improve software on the user's end, do people not base their decisions on benchmarks involving existing (not potential) software? They find comparisons using the provided kernels, find AMD to be slower, unaware that they could (maybe, at that) find a 30% speedup to be had. Even if they stumbled on this article, would they trust they could pull it off, or simply go with the GPU that has the best performance with existing libraries?
These are machines sold for crunching numbers, they might as well crunch numbers as best they can...
The type of commercial logic you're talking about isn't the important factor in the real world. 50% saturation with the option to fully saturate is amazing by AMDs standards and they have much bigger problems than this affecting people's buying decisions. If they had been able to achieve this standard in 2020 I would still be buying AMD.