Ask HN: Best place to learn GPU programing?

If it fits your goals, a useful mental model is that you are programming Numpy ndarray data, and all these tools just fit together to let you manage the data elements, apply transforms, and view them. You should be thinking about the math that solves your domain problem, using alternative mathematical formulations to optimize your algorithms, and only then worrying about using a tuned parallel implementation of that math.

I prototype new ideas and visualize results and intermediate data very easily, with easy transitions between using numpy/scipy routines, custom OpenCL kernels, and OpenGL shaders. I've even found myself using OpenGL to do the visual equivalent of "printf debugging" to look at intermediate array results when developing new array operations and wondering what I've done wrong or misunderstood about the problem. It's very instructive to create little sub-problems you can run as independent scripts during dev/test/microbenchmarking cycles. You should iterate on many small experiments, not assume you can design high-performance solution in one top-down adventure.

For high performance processing, you eventually need to understand architecture limitations and the impact of different problem decompositions. Non-trivial, multi-dimensional problems need to be decomposed into smaller blocks to get better cache locality for the vectorized/parallel code that will process each block. Otherwise, you won't enjoy the benefits of parallel hardware as all the compute units are stalled waiting for data fetches. I find myself doing more and more meta-programming in Python, where I reason about array shapes and sizes, compute slicing geometries, and then loop to extract and dispatch chunks of data to OpenCL jobs which in turn use a dense form of SIMD or SPMD execution on that block. Python is great for this kind of marshaling work.

Also, PyOpenCL makes it trivial to switch between backend drivers. The Intel OpenCL driver is quite good on recent laptop and server CPUs. You might be surprised how much performance you can get out of recent i5/i7 or Xeon E3/E5 CPUs when you use all cores and all SIMD units effectively. Plus, the CPU backend is able to use all system RAM and has a more uniform cache behavior, which makes it more forgiving for badly decomposed problems compared to GPU. This can be a big help for prototyping as well as for limited-run problems where you don't have time to invest in fully tuning your data structures for GPU requirements.

Cautionary tale following...

I don't know the background on this but your goal might not be realistic, unless the rest of the framework architected in a GPU friendly way.

I was once hired for such a project (some scientific applications in chemistry). They had a clear idea what needed to be improved, their Python framework called into C code for some non-trivial matrix operations. This was at the top of the profile so making it faster would have improved the overall performance.

However, each matrix operation was just tens to hundreds of kilobytes in size. Doing them with the GPU was 100x faster, but the latency of getting the data to the GPU and back eliminated any gains.

It's clear that doing these on the GPU would have improved overall performance if the operations could be batched. But the framework was tens of thousands of LOC of scientist-written Python code that was mission critical and could not be rewritten in the time and budget we had.

GPU programming is high throughput at the expense of high latency. It is very difficult to retrofit into a CPU-only framework or application. You need to consider memory management and synchronization for the GPU from ground up.

http://blog.marcgravell.com/2016/05/cudagetting-started-in-n...

For compute, your choices are CUDA, OpenCL, or compute shaders in Vulkan or D3D. CUDA is most likely the best choice if a single vendor API is acceptable.