Gentle introduction to GPUs inner workings

Valve is contributing quite a bit, with their ACO compiler for instance, but the RADV driver did not in fact originate with them. We owe a great deal to David Airlie for that. He wrote a complete vulkan driver years before AMD released the source code for AMDVLK. Valve started contributing a few years in, IIRC.

To AMD's credit, they did help with documentation, questions, and much of the work was built on top of what was there for RadeonSI (OpenGL): shader compiler back-end, etc.

https://www.phoronix.com/scan.php?page=news_item&px=RADV-Rad... -> https://airlied.livejournal.com/81460.html

Note that a lot of the Gallium3D architecture is designed for the stateful APIs like OpenGL. Very little of it is used in the Vulkan drivers.

Yes. That's the idea. Though not exactly sure how the GPU uses the neighboring values to interpolate the partial derivative. It's probably GPU dependent.

That's correct. There aren't enough physical cores to do all cells in parallel. A 1000x1000 grid would have 1M cells. These are virtual computing units. Conceptually there are 1M virtual computing units, one per cell. The thousands of physical cores take on the cells one batch at a time until all of the cells are run. In fact multiple cores can run the code of one cell in parallel. E.g. for-loop is unrolled and each core takes on a different branch of the for-loop. The scheduling of the cores to virtual computing units is like the OS scheduling CPUs to different processes.

Most instructions have no inter-dependency between cells. These instructions can be executed by a core as fast as it can on one computing unit until an inter-dependent instruction like dFdx is encountered that requires syncing. The computing unit is put in a wait state while the core moves on to another one. When all the computing units are sync'ed up at the dFdx instruction, they're then executed by the cores batch by batch.

Pixels are grouped into independent 2x2 blocks called quads and the derivatives are calculated only with respect to the other pixels in the quad.

At the edges of a triangle, when it doesn't cover all four pixels in a quad the pixel shader is still evaluated on all four pixels to calculate derivatives and at the end the results for pixels outside the triangle are thrown away. Your pixel shader better not blow up outside the triangle or you'll get bad derivatives for the pixels that are inside the triangle.

As triangles get smaller, the percentage of pixel shader evaluations that are happening outside of triangles goes up. In the extreme case if the triangle covers only one pixel, the pixel shader is executed 4 times and 3 results are thrown out! This has started to become a problem for games with high geometric detail and it's one of the reasons why Epic started using software rasterization for their Nanite virtualized geometry system in Unreal 5.

You might say "why don't you just stop doing this approximate derivative thing so you don't have to execute pixels that are thrown away? Are derivatives that important?" The answer is that derivatives are used by the hardware during texture sampling to select the correct mip level, which is pretty much essential for performance and eliminating aliasing. So you'd have to replace that with manual calculation of derivatives by the application which would be a big change.

> Do you have a hardcore introduction with even more detail / perhaps even with examples of implementations?

Hard to beat the actual technical references when you want something hardcore!

The actual CUDA documentation is good:

https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....

In particular, "Section 5: Performance Guidelines": https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.... gives a lot of micro-architectural details, including those nasty "bank conflicts" that people keep talking about. Honestly, the original documentation says it best with just a few paragraphs on that particular manner:

> To achieve high bandwidth, shared memory is divided into equally-sized memory modules, called banks, which can be accessed simultaneously. Any memory read or write request made of n addresses that fall in n distinct memory banks can therefore be serviced simultaneously, yielding an overall bandwidth that is n times as high as the bandwidth of a single module.

>

> However, if two addresses of a memory request fall in the same memory bank, there is a bank conflict and the access has to be serialized. The hardware splits a memory request with bank conflicts into as many separate conflict-free requests as necessary, decreasing throughput by a factor equal to the number of separate memory requests. If the number of separate memory requests is n, the initial memory request is said to cause n-way bank conflicts.

See? Its really not that hard or unapproachable. Just read the original docs, its all there.

AMD's documentation is scattered to the winds, but the same information is around. I'd say that your #1 performance guidelines are from the ancient optimization guide from 2015. Its a bit dated, but its fine: http://developer.amd.com/wordpress/media/2013/12/AMD_OpenCL_...

Chapter 1 and Chapter 2 are relevant to today's architectures (even RDNA, even though some details have changed). Chapter 2: GCN, applies for all AMD GPUs from the 7xxx series, through the Rx 2xx series, Rx 3xx, 4xx, 5xx, Vega, and CDNA (aka: MI100) architectures.

RDNA does not have as good of an architectural guide. Start with the OpenCL guide for optimization, and then "update" your knowledge with the rather short RDNA guide: https://gpuopen.com/performance/

For assembly language details:

* CUDA's PTX is basically assembly language: https://docs.nvidia.com/cuda/parallel-thread-execution/index...

* PTX is a portable assembly language: NVidia continuously updates their GPUs. Volta was very well studied by this paper: https://arxiv.org/abs/1804.06826. I'd suggest reading this AFTER you learn the basics of PTX.

* AMD publishes their assembly language for each generation. Vega is probably a good starting point: https://developer.amd.com/wp-content/resources/Vega_Shader_I...

Note that GPU.js is not for graphics, it's primarily for doing GPU Compute. I'd recommend looking at a 3D scene library to get your feet wet (and there are plenty of those in JS), or if you're interested in the underlying workings, to look at either WebGL or WebGPU.

In AMD land, `rocm-smi` is like `top` for your gpus. Very similar in capabilities and functionality to NVIDIA.

> 2D graphics is very different and mostly doesn't require the GPU's assistance.

There are interesting tasks in 2D graphics that may easily warrant GPU acceleration, such as smooth animation of composited surfaces (even something as simple as displaying a mouse pointer falls under this, and is accelerated in most systems) or rendering complex shapes including text.

Eh, it's sort of a naming conflict.

There were two dimensions of "vector" -- whether you used XMM-style registers, requiring the compiler to auto-vectorize different operations. The old vector ISAs had things like dot product instructions. Those things are now all gone, because it turns out it's hard to optimize.

We in the industry call scalar-for-each-thread ISAs as "scalar ISAs" these days. For instance, Mali describes their transition from a vec4-for-each-thread-based ISA to a scalar-for-each-thread-based ISA as transitioning from "vector to scalar" [0].

[0] Compare 4:26 and 6:29 in the "Mali GPU Family" video on this page https://developer.arm.com/solutions/graphics-and-gaming/arm-...