Too bad it's CUDA
Sooner or later this will become a problem because you are depending on the benevolence of a single manufacturer.
Compilers generate PTX, then the rest of the CUDA infrastructure turns PTX into Turing machine code, or Ampere machine code, or Pascal machine code.
In theory, SPIR-V should do the same job, but its just not as usable right now. In the meantime, getting it to work on PTX is easier, and then there's probably hope (in the far future) to move to SPIR-V if that ever actually takes off.
I'm not a developer on Triton, but that'd be my expectation.
Khronos mindset of it must be C like and parterns will take care of the ecosystem is what doomed OpenCL.
All of their API design endevours are "design by committe" at its best.
No wonder that SYSCL is now backend agnostic.
There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions)
---------
I think CUDA vs OpenCL is over. ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL. Its a superior model: having a single language that supports both CPU and GPU code is just easier than switching between C++ and OpenCL (where data-structures can't be shared as easily).
-----------
The main issue with AMD is that they're cutting support for their older cards. The cheapest card you can get that supports ROCm is Vega56 now... otherwise you're basically expected to go for the expensive MI-line (MI50, MI100).
Vulkan / OpenCL / etc. etc. don't have any data-structure sharing like that with the host code. Its a point of contention that makes anything more complicated than a 3-dimensional array hard to share.
Yeah, Vulkan / OpenCL have all sorts of pointer-sharing arrangements (Shared Virtual Memory) or whatnot. But its difficult to use in practice, because they keep the concepts of "GPU" code separate from "CPU" code.
---------
When you look at these things: such as Triton, you see that people want to unify the CPU-and-GPU code into a single code base. Look and read these Triton examples: they're just Python code, inside of the rest of CPU-Python code.
I think people are realizing that high-level code can flow between the two execution units (CPU or GPU) without changing the high level language. The compiler works hard to generate code for both systems, but its better for the compiler to work rather than the programmer to work on integration.
Yes and no. You find surprising bits of code that can be shared.
Read through a high-performance GPU project like "GPU perft" (https://github.com/ankan-ban/perft_gpu/blob/master/chess.h). The "perft" problem is the problem of counting the number of board positions reachable in X moves. (Perft(15) is "how many board positions exist within 15 moves from the start of chess?". Perft(1) is 20, and Perft(2) is 400, then 3+ gets a bit complicated!).
You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU.
In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation.
------------
The actual search (how to branch off, coalesce results, coordinate threads) is extremely different between CPU and GPU. So of course, you want that code to be written in a CPU-specific, or GPU-specific manner.
But the question of "where can this Bishop move??" (the sliding piece attack subroutine) is identical on CPU or GPU.
I just wonder if this alone is enough to make a unified programming environment the preferred way. On the web we had Meteor.js which tried this approach to unify client and server with javascript. The shared code was of similar types. Meteor never became the preferred way to write web apps. Im wondering if the same is true for GPU + CPU programming.
In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem.
CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked but had performance impact. But they were widely used because it was too hard to do yourself.
Even in video games, where GPUs do graphics and CPUs do physics, the lines have been blurred with TressFX (hair physics on GPU instead of CPU) or PhysX (collision physics done on GPU instead of CPU).
GPUs are general purpose computers now. You can program them to do whatever you want. So when the programmer is writing code, it pays to experiment a little: maybe run the code on the GPU... see if the GPU really is slower or faster. You might be surprised.
Ditto on the reverse: maybe some bit of GPU code would be better run on the CPU.
--------
This guess-and-check, program-and-profile methodology is the same as any other high-performance code people write. You throw away assumptions and just test the heck out of all your ideas, because you're surprised far more often than not in this field.
As such, having the ability to quickly move code from GPU-side to CPU-side (and vice versa) for these tests is extremely beneficial. You don't really know which device will run your code faster until you actually test it. (Though your instincts get better the more experience you gain)
GPUs are (basically) hardware devices for doing vast numbers of matrix manipulations very very fast in parallel, with limitations on the size of the matrix.
CPUs can do matrix manipulations, with less parallelism.
Things like TressFX uses the tridiagonal matrix algorithm[1] to solve a linear system representing the hair constraints system. This could be implemented on a CPU as well, doing exactly the same thing with the same algorithm. It would run slower though!
[1] https://en.wikipedia.org/wiki/Tridiagonal_matrix_algorithm
[2] Slide 37, https://www.slideshare.net/DevCentralAMD/gs4147-billbilodeau
You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU.
In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation.
The ceremony around co-ordinating threads and memory access is different, but the code that is being run is exactly the same.
For example, in neural networks the key algorithm is differentiation. That is implemented the same way on both CPUs and GPUs.
No, their main issue is not properly supporting ROCm in general. No Windows support at all? It still feels like they don't know whether they want to continue investing in ROCm long term.
That really not the point. OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors. ROCm is AMD-only, and even that is questionable since it didn't exist 6 or 7 years ago, and who knows - they might drop it like they changed their earlier focus.
Also, CUDA has a much richer ecosystem than AMD ROCm (I'm sad to say; as I'm not a fan of NVIDIA).
As was C++AMP (which was actually pretty good IMO as a language). Just because its a standard doesn't mean its going to be used.
OpenCL 2.0 was very poorly implemented: almost no one used any of its advanced features. To the point that OpenCL 3.0 is resetting from OpenCL 1.2.
Only Intel really supported OpenCL 2.1 or OpenCL 2.2. The entire OpenCL 2.x branch for years was squandered with tepid responses from NVidia and AMD (yes, AMD had better OpenCL 2.0 support. But it's debugger didn't work, all of the code was tested on OpenCL 1.2 only. No one cared)
I dare say that OpenCL 2.x was about as "standardized" and respected as C++ AMP. Just because its an open standard doesn't mean that its actually useful. Any serious OpenCL programmer stuck with OpenCL 1.2, including both AMD and NVidia OpenCL programmers.
Still, that doesn't contradict what I said in my earlier post.
There was also OpenCL C++, which I really wanted to see implemented by GPU vendors, especially NVIDIA, and that didn't happen either.
As for C++AMP - did that ever work on NVIDIA hardware?
ROCm isn't a programming language or programming model - it's a collection of tools and libraries. Which language are you specifically referencing here? HIP?
And the less said about Intel it's better. Smart, motivated people I know have left Argonne recently because of the Aurora shitshow.
Intel, last I heard has written off close to 300 million dollars on Aurora.
I mean, I have the gist. Intel bet big on Xeon Phi, but that didn't seem to offer enough performance. Intel then switched over to this GPU-strategy (see Intel Xe), but that required them to rework virtually everything from scratch.
No one said anything in my previous paragraph. But its blatantly obvious: the Xeon Phi was being advertised very strongly by Intel up to the point that Aurora's design was reworked. Suddenly, Xeon Phi was cut, and Intel started talking about Xe (including OneAPI, and all sorts of stuff the HPC market cares about). I'm confident enough at reading between the lines that I'm willing to assume this in the absence of hard facts :-)
I'm kind of curious on the play by play, if at all possible. What test showed up that made Argonne National Laboratory decide that the Xeon Phi model wasn't going to work? Was it possible to come to this conclusion any sooner? I realize this sort of stuff is rarely published, but I guess that's what makes me interested in those juicy details.
Secondly, DOE has a strategy (mostly rightly in my opinion) of not putting all its' computation eggs in a single companies' basket. Thus national labs compete between themselves as do companies. Notice the cadence of computing purchases: IBM/Nvidia -> Intel/Intel -> AMD/AMD.
Thirdly, Intel had promised that using OneAPI existing GPU optimized simulation codes could be translated to Intel GPUs with minimum effort. The idea had merit, back in 2016-2017. I have a CUDA based simulation code - it's almost a matter of recompiling to Intel, with some minimal effort on my part. That it didn't work out - well hindsight is 20/20
>Under development: AMD GPUs, CPUs
Also, if we only cared about open standard we'd simply not use a GPU at all and settle RISC-V chips :-)
NVidia is faster at 4x4 16-bit matrix multiplications (common in Tensor / Deep Learning stuff), but MI100 still has 4x4 16-bit matrix multiplication instructions and acceleration. Its not far behind, and the greater 64-bit FLOPs is enough to win in scientific fields.
El Capitan (Lawrence Rivermore National Labs) is MI100 and specified as such. So I'll "retreat" my argument to that more easily verified position. Still a top supercomputer in the world, and more is known about El Capitan than Frontier.
It's possibly worth saying that these things presumably are competitive with nvlink-based multi-GPU, and appear to support unified memory, but I don't know how that compares with ac922s. Not that UM is relevant for what I know of usage on our system....
I have plenty of links, but they're all from non-primary source newsites, like servethehome or (worse) wcftech. Those are more "casual" blogs that aren't worth quoting IMO and are closer to rumormills (even if they sometimes have good information).
I can't find a single spec from AMD's website or ORNL, or the Department of Energy (primary sources: the ones we know would tell the truth on this matter).
The closest I got is this: https://www.amd.com/system/files/documents/oak-ridge-nationa...
Which is an MI100 test that ORNLs did in "preparation" for Frontier. It doesn't seem like Frontier nor El Capitan have had their specs officially released yet.