Rust-GPU

> it turns out to be really hard

Everyone was yelling this at you since the beginning. Ever since Apple proposed "safe HLSL", and then it morphed into WHLSL and then WLSL and then the competing Tint proposal which is now WGSL, everyone (Adobe, Valve, Epic, Unity, NVIDIA) was saying "none of this will do what you think it does, we've all been here before and it's very very hard, at the very least, get some IHVs or at least a team that does port work / emulation tech on board".

The committee chose not to listen, instead forcing WebGPU down this hell that nobody really wants or appreciates. The whole WebGPU is three years late and still has no users, despite being one of the most complicated specs to land in years. And yeah, I'm still a bit bitter about it.

I am also skeptical of rust-gpu's approach, because I don't think it's possible to take a CPU-inspired language design and blindly adapt it to GPUs without some difficult questions to answer (they still don't have flow control, but I'm told that's coming).

But at the very least, I think repi and Embark know what they're doing and know about the tough problems, rather than feigning being blindsighted.

I'm going to guess that it's a subset of Rust.

Rust has hygienic macros and a package manager, at least. It'd also be nice to have a common language that you could run on the CPU for debugging purposes. I see this more as a moonshot type project though, where if you could accomplish it, it would mostly just speed up Rust's time to utility for other similar use cases like embedded development.

Another advantage (for Embark) is their main engine codebase is also in Rust. This allows for sharing of some code on GPU and CPU. This isn't uncommon in the game industry, for example header files that compile in both C++ and HLSL in order to share struct definitions for CPU code and GPU code, and CPU math libraries that use HLSL types and semantics.

Rust is different enough from GLSL and HLSL (most game engines use HLSL as "native" and cross compile to GLSL, MSL or use header defines to build PSSL) that introducing a new shading language compatible with Rust is probably worth the effort for them.

The main benefit is that the GPU-side code can better integrate with the CPU-side code, and that you don't need separate tools and a complex build process integration.

For instance it's possible to share data structure between the CPU and GPU side directly from the same source code, or you can run the same code on the CPU or GPU for whatever reason (debugging, unit testing etc...).

The Metal API uses C++ (with some extensions) as shader language, which IMHO is complete overkill too, but one nice side effect is that you can put shared data structures into headers, and then include the same shared headers on the CPU and GPU side.

Ideally it should be possible to mix GPU and CPU code in the same source file, and only (for instance) annotate GPU functions with an attribute which causes it to be compiled as shader code.

From the readme

Focus on Vulkan graphics shaders first, then after Vulkan compute shaders

So no compute support yet. Also, Rust-GPU's focus is writing shaders in Rust, in order to support a game engine, so the focus will be real-time graphics and simulation. It seems like it will never evolve into a CUDA like heterogeneous environment that's more common in scientific computing and ML.

The project actually says it compiles to SPIR-V (Vulkan's standard intermediate representation), so driver/GPU support really shouldn't be too much of an issue so long as the target is one that supports Vulkan.

> Plus, not all engines have the same data layout, so an “effect crate” would either require your engine to work a very specific way, or would require integration work that would probably defeat the purpose of using a package manager in the first place.

You could say the same thing about nodes in shader graph systems, but in practice they work quite well.

> It can’t be worth the headaches, especially when you consider how a TON of devices still only support OpenGL, which not only requires runtime compilation of shaders (so only GLSL unless your app ships a cross compiler)

You can AOT compile SPIR-V to GLSL with SPIRV-Cross.

CPU code calls in to drivers which talk to the GPU over the PCIe bus. So assembly code that does stuff with the GPU just looks like loading a shared library, passing in some shader code, and then passing in some drawing commands. This project looks like it aims to compile rust to a shader intermediate format so you can use it instead of more common shading languages like GLSL or HLSL. Basically your HLSL/GLSL/(now Rust) code ends up as an intermediate format, which the CPU passes in to a graphics driver before asking that driver to draw things.

CPU peripherals are memory, so it just looks like copying a GPU "kernel" (program) from a memory area (from disk) to another (GPU, via PCIe).

I don't know enough about PCIe to be an authoritative source, but I'll just go along and describe the way most microcontrollers and peripherals work. I doubt x86 has instructions dedicated to PCIe. There will likely be a PCIe controller memory-mapped to some hardware address, that the OS will write to as if it was regular memory (that address is physically wired to the controller, be it on the CPU die or on the motherboard). The address is generally specified by the BIOS/UEFI (devicetrees on most embedded platforms).

Of course, what you will write will depend on the controller itself, but the OS will have a driver that contains that information. You have a back-and-forth to list peripherals on the bus, identify the one you want to communicate to, set-up access mode (direct memory access where you map an area of your memory to the peripheral, or just sequential access trough the controller). Once you can communicate with the GPU, you start the same dance to give it a program you compiled ahead of time, have it run, and fetch the result.

I don't have low-level experience with that specific part of the system, but that kind of stuff is generic enough that you find the same patterns everywhere, at least for turing machines, which x86 CPUs are.

It's not (currently) an instruction. The key concepts at the hardware level are buffers, descriptors, and doorbell registers. These are accessed using mov instructions.

https://compas.cs.stonybrook.edu/~nhonarmand/courses/sp17/cs...

The OS level is purely so that multiple programs can talk to the same hardware at the same time. For example, your Firefox Browser may talk to the GPU to render something (H.264 video), or to send audio over the HDMI cable. But at the same time, you're playing some video game (Civ-6), and the OS needs to prioritize which gets access.

------------

When it comes to I/O, there are two strategies: memory mapped I/O, and I/O ports (the "in" and "out" assembly instructions). I'm pretty sure most modern hardware is memory-mapped type, where you just read to (or write from) certain "memory" locations.

Its not really memory (it doesn't go to RAM), but instead to some other component, like PCIe root complex for further processing.

------

Something to note: the whole OSI level exists on the motherboard. You've got physical connections, you've got link-layer connections, and even a network similar to TCP/IP. In fact, there are multiple networks: SATA networks, PCIe networks, and USB Networks, that have their own addressing rules and physical protocols. Different USB Ports, different PCIe addresses, and more!

From the perspective of the CPU, I expect any "GPU Command" to simply be a link-level call to the PCIe root complex. You send data to the PCIe controller, which then forwards the data to the GPU. Or NVMe RAM, or your network card.

To fully make a GPU Command get to the GPU, you need to know the PCIe address, and route the message appropriately (there could be PCIe switches in between). Finally, there's a protocol, similar to application-level code (HTTP) where applications talk to the GPU.

Given the similarities between DirectX, OpenCL, Vulkan, Metal, and CUDA/HIP, I expect that GPUs all have roughly the same "application" interface. Shaders are compiled into machine code. Machine code is loaded into GPU VRAM. Command Queues issue remote-function calls with an event-driven dependency graph that the GPU selects kernels from. Etc. etc.

This is a bit of an oversimplification, because I don't actually know the details, but it works roughly like this: GPUs are devices on the PCI Express bus, without an operating system getting in the way everything on the PCI Express bus is organised into something called ECAM (https://en.wikipedia.org/wiki/PCI_configuration_space). Each device there specifies memory regions it supports. These memory regions map to a large number of configuration registers and on-device memory the GPU supports. Since GPUs are really complex most of its behaviour is actually not directly controlled but instead predetermined by its firmware (basically an operating system in its own right). Finally GPUs really execute multiple instruction streams with proprietary instruction formats and can independently load data from main memory with their build in DMA engines. These operations are typically kicked off by pointing various registers on the GPU to memory regions in main memory to tell it where to fetch instructions and data from. Everything else happens for the most part as consequences of instructions executing on the GPU.

This specific component is a SPIR-V backend for Rust. In practice, how you use this is you use a native API (Vulkan), give it your generated SPIR-V program, and it "runs it" using its own semantics. This is a combination of a userspace API talking to a kernel driver, and the kernel driver using a combination of IN/OUT instructions (unlikely) and memory-mapped I/O (much more likely) to talk to the GPU.

The actual details of the communication is documented by some IHVs, either through PDFs or source code. Normally, the CPU doesn't wait for programs to complete, but instead, the GPU driver has a list of tasks to run (a "command buffer") that it itself can schedule across many different pieces of hardware, and a combination of the driver and GPU itself determine scheduling, execution, and so on. Note that there's still a lot of work for the driver to do, including managing the device's memory (textures, buffers, so on), compiling any programs into machine language (similar to a JIT), and coordinating multiple different programs trying to use the GPU.

> Or is it at the OS layer instead of the assembly layer, with an OS syscall that defers to the GPU?

Yes. Modern OSes don’t allow userspace code to directly access peripheral devices, GPUs included. Due to the enormous complexity of modern GPUs (GPUs typically have more transistors, consume more electricity, and have comparable amount of memory), the API surface is huge.

On Windows, the native API surface is Direct3D. It has two large pieces, user-mode and kernel mode. The vendor-provided GPU driver is similarly split in two.

User-mode parts implement HLSL compiler (Microsoft), another downstream compiler to compile DXBC into proprietary byte code GPUs actually run (vendors), implement other higher-level stuff like mipmap generation (vendors, at least for nVidia), and expose user-facing APIs, i.e. multiple versions of D3D (Microsoft).

Kernel mode driver interfaces with actual hardware, the Microsoft’s vendor-agnostic part of that is in dxgkrnl.sys.

Various GPU drivers often expose extra user-facing APIs in addition to Direct3D (CUDA, Vulkan, OpenGL, OpenCL), but all of them are optional.

Simplifying a lot, both the CPU and the GPU have access to a few regions of shared memory, which both can read from and write to. Some of it is on the GPU (device local memory) and some of it is on the CPU (host local memory). The CPU writes a list of commands to one of these shared memory regions, then writes to a special register on the GPU (these GPU registers are visible to the CPU as yet another region of shared memory) telling it where that list of commands is.

That is, from a x86-64 assembly point of view, it's just writing to memory. Usually, only the final write to the special register is privileged, so that write will need an OS syscall into the graphics driver; the rest can be written directly by the userspace driver (which used beforehand another OS syscall to get access to some of that memory shared with the GPU). The userspace driver also compiles the SPIR-V programs into native GPU code, which can be loaded and executed as instructed by the list of commands.

From the link: "Focus on Vulkan graphics shaders first, then after Vulkan compute shaders"

> Can Rust be used also for compute?

It can, but it's worth noticing that the programming paradigm is different. For this reason, I would assume you are still writing a DSL within Rust which can use Rust's type checker to help you figure out what to write and how, but Rust in itself cannot represent the same semantics as-is that is required to write efficient parallel code. So, you still need to be a domain expert. But, automating the bytecode writing is alleviated by Rust's ability to provide packages etc.

I refactored some OpenMP code to run on SPIR-V using the Vulkan API the OP project uses, and I must say that I cannot see myself doing it in any other language than Rust.