GPU Programming: When, Why and How?
enccs.github.io
enccs.github.io
Disclaimer: I've recently finished a PhD focused on memory optimizations in Futhark.
[0]: https://futhark-lang.org/
Edit: They do actually mention stuff like Julia and NumPy.
It's absolutely a much lower barrier to entry, especially if you are familiar with functional programming. However, it also leaves a lot of performance on the table, and I found it to be hard or cumbersome to integrate with languages other than Python at the time.
And you're still stuck with CUDA (which is single vendor) or OpenCL (which is a pain to set up on most consumer systems, and has very lackluster driver quality for a lot of vendors). I would have liked that by now support for Vulkan compute or OpenGL compute shaders would have been added, but alas.
Regarding the performance, I'd be interested to hear more about your experiences. Because we compile to CUDA or OpenCL in the end, we cannot claim to be faster than what you could (in principle) write in hand. However, most of our benchmarks compare favorably with handwritten reference implementations, and the heavily optimizing compiler is able to write code that is tough to write by hand.
However, we're always looking for instances where we can do better. The goal is to be comparable to CUDA/OpenCL in as many cases as possible.
Like I said in my earlier comment, it was so long ago the performance remarks may be outdated and the project may have resolved them.
I would like to give Futhark another try, but I need easy and portable integration in either Go or Rust, and as far as I can tell, that's not the case today? For my current project, I ended up selecting webgpu for my GPU compute needs.
* Futhark does not expose a scheduling language that gives you precise control over code generation. This is probably the main selling point of Halide.
* Futhark has a much broader focus than Halide, which is mainly oriented towards image processing. Futhark wants to support arbitrary data parallel computation. E.g. see this compiler written in Futhark: https://github.com/Snektron/pareas
Compared to Sycl:
* Futhark is a non-embedded language that is more high level than Sycl. The goals are similar in the sense that both systems to try make (data) parallel programming more accessible. The vision behind Futhark is that the conventional functional programming vocabulary is actually a pretty good fit for parallelism, and that an aggressively optimising compiler can reduce or eliminate the overhead of abstraction. I don't think Sycl is as focused on high levels of abstraction, but rather focuses on being a relatively low-level portable programming interface.
A more simple way to get into GPU programming, would be WebGPU.
I can recommend this tutorial as an introduction:
https://surma.dev/things/webgpu/
GPU programming is really a different way of programming, even though the code looks mostly familiar. And if you are used to single threaded, it will give you a massive headache thinking about, how to do everything in parallel, so that everything is not blocking everything.
The raw power is tempting, so much that I evaluate now how to do allmost everything on the GPU, but many things simply are not suitable to do there.
Basically, you want to avoid conditions and loops. Cannot do recursion. Debugging is a pain and non obvious things can make everything very slow, until you roughly understand, what is going on inside.
I found it was worse than that when I started playing with shaders, because my idea of parallelism came from threads, workers, and async functions eg doing several separate tasks at the same time. Parallel on a GPU is more like running lots of the same function each with different parameters at the same time eg running a function to calculate the color of a pixel using its x and y coordinates (and maybe some other uniforms) for every pixel on the screen.
You give it a bunch of data with elements which are all independent of each other, apply the same transformations to to each element, and output a single data structure (a colour attachment or render target, or some compute buffer).
This explains a lot of the shortcomings and design decisions of GPUs. They prioritise throughput over latency; they have a lot of memory but much slower than the cores themselves (~500 – 700 cycle latency compared to ~20 – 40 cycle latency on the CPU); they run 'threads' in synchronised lock-step, called warps; branching on a GPU is particularly expensive—if one shader for one fragment needs to branch, then the entire warp that it's on will have to wait for it, or worse still, calculate that branch, too;
Their 3D graphics legacy still endures: despite the advent of GPGPU, they still have plenty of dedicated hardware for the graphics pipeline, including rasterisers, texture-mapping units, render output processors, etc.
Once people get this idea, then GPU programming is easier to reason about. One execution of a shader deals with one fragment (actually four, but at a high level of abstraction, this is sufficient).
Oddly enough, I did GPU parallelisation first, and then came to CPU parallelisation. I find CPU-level parallelism and concurrency problems like OS inter-process synchronisation, async-await, etc. much harder to reason about, because they generally cannot be reduced to such a simplistic 'one thread for one pixel'. It's also why I like embarrassingly parallel problems, because they're the most easily GPU-able.
*techinically it's even true for CPU in some cases...
Practically in all cases. It isn't as bad to branch as it is in a GPU. But branch prediction in CPUs is an important factor for performance critical code, and getting it right (e.g. by switching the 'branch/no branch' conditions in assembler or using branch hints such as likely/unlikely in the linux kernel) can make a very noticable difference. I've seen a factor of 2 in some tight loops, and 10% should often be achievable.
And sometimes, by using bitmasks or multiplication by 1/0, one can design branchless code which can enable the use of e.g. more AVX instructions on CPUs, while also avoiding branch miss penalties.
Conditions and loops are largely fine as long as you avoid warp divergence and such.
https://google.github.io/tour-of-wgsl/
No need to setup, install or config anything. I think that is arguably easier as a quick start.
Otherwise yes, the whole tooling and everything else is not really that mature in WebGPU.
And thanks for Taichi. I did not know of that libary and it sounds interesting! (I will stay with WebGPU for now)
Yes you can! (because of WebGPU)
And I never said WebGPU is high level. I said and meant it is simpler to get started with. At least it was quite simple for me. But WebGPU and wgsl by itself are surely not "simple".
Learned that computation should be highly local (working on small isolated pieces of data) and without if's, otherwise won't get any performance benefits.
So if some threads need to take an IF-branch, and some need to take an ELSE-branch, both paths are executed in sequence, with all threads in the group executing in lock-step, and a per-thread mask disabling the effects of the instructions that aren't needed for each specific thread.
If you do this enough times, with sufficiently complicated branching, you'll quickly negate any efficiency gain you get from the massive parallelism.
To simplify: Each 128 SMs can run a different program and each of their 128 "cuda cores" can do ~single-cycle (small) matrix operations.
My mental model is a bit more like you have collections of warps in a block, and all warps in a block get scheduled onto an SM. Different GPU architectures allow for different numbers of warps to be simultaneously active or inactive, and each warp has its own instruction pointer and can be suspended while waiting for things like memory. I found the picture on pg 22 here really helpful: https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente...
Note that although there's 4 schedulers, on the A100, they don't dispatch every cycle iirc.
Correct me if I'm wrong, but as far as I can tell tensor cores are just accelerators. They can't do general compute: no branch or jump.
Like you said, tensor core is similar to a special purpose ALU and is at a lower level of abstraction than something with an instruction pointer.
Everything is on the device during the tight loop, and there is minimal transfer to host, so I don't think it's bandwidth limited in that sense. It could be bandwidth limited on the cache level but I couldn't tell. I wish I could get Compute running.
Only when that does not already cover all your needs you have to find or develop other software for handling sparse matrices on GPUs.