How do you get started with learning GPU programming? I mean that in a pedagogical way, like what is the best educational way forward. Build stupid stuff in shader toys?
I think there are probably a number of on-ramps. One easy way to get started is shadertoy, which if nothing else should give familiarity with GLSL syntax and intuition for performance in the simplest case: each "thread" computes something (a pixel value) independently of the others. You'll quickly run into limitations though, as it can't really do any of the more advanced compute stuff. I think as WebGPU becomes real, an analogous tool that unlocks compute kernels (aka compute shaders) could be very powerful for pedagogy.
I think most people that do compute on GPU use CUDA, as it's the only really practical toolchain for it. That has a large number of high quality open source libraries, which tend to be well-documented and have good research papers behind them. You can start by using these libraries, then digging deeper to see how they're implemented.
As I've been going on about, I believe this space is ripe for major growth in the next decade or so. As a rough guide, if you can make your algorithm fit the GPU compute model nicely, you'll get about 10x the compute per dollar, which is effectively the same as compute per watt. Why leave such performance on the table? The answer is that programming GPUs is just too hard. In certain areas, including machine learning, an investment of research has partially fixed that problem. But in others there is fruit at medium height, ripe for the picking. And in some other areas, you'll need a tall ladder, of which I believe a solid understanding of monoids is but one rung.
Most universities don't really teach parallel algorithm development in any way despite the core count increasing in both CPU and GPU world.
Current languages like GLSL and CUDA are very abstracted in the sense that they don't really tell why some approach is better or worse than the other. To me, if you can create as general cellular automata -like solution, then such logic can be converted and is generally fine performance-wise for SIMD orientated programming.
I made this [1] simple shader because I couldn't find the effect in Open-Source video software...