Hacker News Reader

Comment by kypro | Hacker News Reader

As someone who's very recently been looking into using WebGPU on a project, why must it be so complicated to execute some simple instructions on a GPU?

All of the code I was looking at seems to use (and require from the API) a lot of graphics terminology like "shaders", etc. Additionally it seemed to require a bizarre amount of boilerplate code to create compute "pipelines" and other things.

Is there some reason that running code on a GPU must be so difficult? Why is there no API I can call like,

`GPU.exec(() => 2 * 2);`

Sorry for the dumb question. I'll definitely be taking a look through this guide later.

View on HN

Hi, I'm a graphics programmer. Sorry for our confusing and pretty awful terminology, it's mostly an accident of history. I don't think we're the only field to have that problem, though.

> why must it be so complicated to execute some simple instructions on a GPU

Because they're not simple instructions. Here's what your 2 + 2 compiles to on an AMD GPU: https://hlsl.godbolt.org/z/qfz1TjPGa

In order to fill a super wide machine, we need to give it enough work to fill. That means we need to schedule a large number of jobs, and we those jobs to have a hierarchy. The hierarchy for compute work is:

Dispatch -> Workgroup (aka Thread Group) -> Subgroup (aka Wave, Wavefront, Warp, SIMD Group) -> Thread (aka Wave Lane, Lane, SIMD Lane)

Much like any weird device like a DSP, there's a lot of power here, but you need to program the hardware a certain way, or you'll get screwed. If you want to be immersed in what this looks like at a high level, go read this post for some of the flavor of the problems we deal with: https://tangentvector.wordpress.com/2013/04/12/a-digression-...

GPU parallelism isn't magic, and a magic compiler can get us close, but not all the way.

As a fellow graphics programmer, this is a perfect answer.

Are you the guy that made the link cell shading video? If so, I really like your stuff! Any advice on how to get into graphics programming?

I am, yes! Thank you!

I'm actually been a spec contributor to WebGPU, and can vouch for quite a bit of it, I think it's a pretty great API overall, and I think it strikes a nice balance between the apparent-simplicity-hidden-complexity of OpenGL and something like daunting like Direct3D or Vulkan.

A lot of the things I mentioned above don't really come into play for quite some time. You can get quite far with just triangles, vertex shaders, and pixel shaders. So go give it a try, draw some stuff, and then work your way up and down the stack.

Any thoughts on mesh shaders?

From what I've seen (as a non-graphics programmer), the paradigm seems closer to what I'm used to.

Though I'm guessing that limited hardware support will hamper adoption.

Oh wow lol. I know we've gotten into discussions (probably some debates) on HN before. Never connected your name to that channel. I'm apparently subscribed and have seen 5 of your videos so far. All really fantastic videos. I apparently missed the Mario Galaxy 2 video, so there goes the next hour of my evening.

Aside from the other comments which provide great answers about the way GPUs work and their history... I want to give my gut instinct for why an API like this doesn't exist on top of all that history.

I suspect it's simply because the "complexity floor" for running code on a GPU is already high. Obviously your trivial example is only for the sake of example. But I suspect that nobody has had the incentive to create an API that looks like that because anything close to that is just better done on the CPU.

Once you have problems that need a GPU to solve, then working through the peculiarities of the complex API is worth it.

I don't know this for a fact, but it's me reasoning backwards from the fact that nobody has seen fit to create an API like this, when all sorts of zany APIs exist.

For example, there are already plenty of code generators for graphics-oriented shader programs, which will take e.g. a material description like `{color: red, opacity: 0.5, reflective: true}` and generate the GLSL code.

Why would there not be an equivalent which compiles an expression language to GPGPU instructions? I must assume it's because the programs you might want to express with it are probably best run on the CPU.

My example wasn't great, and you're right that this would be better ran on a CPU.

For context, my very naive working model of GPUs is that they're basically just massively parallel CPUs with slower clock speeds. So given this it seems like it would make sense to have some kind of async math library which allow you to send batch calculations to the GPU to run in parallel.

Basically anything where you typically create a loop and do some calculation seems like it would better batched and sent to the GPU. Assuming my working model of GPUs is correct anyway.

So last week I was working on a JS library which required doing exactly this – some basic addition and multiplication in a loop – and I thought, hey, I wonder if I can just batch this and send it to the GPU for a slight performance boost? I was aware of projects like GPU.js and figured it should be a relatively simple thing to try, but turns out I was wrong.

What I thought would take me 30 mins last week has now turned into a quest to learn about shader code and the history of GPUs, and I'm still not exactly sure why.

"basically just massively parallel CPUs" is correct-ish, but it has the added wrinkle that you have to think of them as CPUs where large chunks of cores have to be executing the same instruction at the same time, which means that it's mostly loops where each iteration runs close to the same instructions that make sense.

Also, I think you might be drastically underestimating how utterly insane modern JavaScript engines like V8 are. JS code running as well as it does on CPUs is sort of a minor miracle in the first place. Even then, there's still a place/need for WebASM on top of it.

Transposing that unto a GPU in a transparent manner is asking a LOT. Not to mention that there's a lot of stuff you can do in JS that, if feasible at all, makes no sense whatsoever in GPU-land. Even basic stuff, e.g. string-based property lookups, just does not belong in GPU code for the most part.

If I see an API that looks like `GPU.exec(Callable)`, then I expect that any callable can be passed to that API. In JavaScript, that can mean a LOT of different things, many of which would be hell on earth to convert to GPU-compatible code.

Very, very wide SIMD units is also a good model. Your performance is proportional to how much of the same thing you're doing at once

TIL about GPU.js, which on the face of it seems like exactly what you were asking for in your original comment. Was the problem that the code you wanted to write didn't quite fit into the expected execution model, even if GPU.js allowed you to write things in an ergonomic way?

Many reasons, some of them not technical (e.g. historical baggage and closely guarded IPs), but mainly because GPUs are by far not as standardized and "open" as CPUs. On CPUs, the instruction set is basically the abstraction layer to the underlying hardware, on GPUs the abstraction layer is a software driver instead, and that needs to be much more abstract because the differences between the various GPU architectures are much bigger than the differences between ISAs like x86 vs ARM.

Having said that: your idea absolutely makes sense, and I would love to see shader compilers merged into regular compiler toolchains, so that CPU and GPU code can be easily mixed in the same source file (technically it's definitely possible, but this would also be the death of "3D APIs as we know them" - those would basically be merged into language stdlibs, and it would require many parties working closely together - so there's a risk of the 'many cooks' problem).

But give it a decade or so, and hopefully we'll get exactly that :)

> I would love to see shader compilers merged into regular compiler toolchains

they are already using the same compiler infrastructure: pretty much every relevant shader or otherwise GPU oriented compiler out there is based on LLVM. As for the parts of the toolchains that are different: that's simply because GPUs and CPUs are inherently very different.

> so that CPU and GPU code can be easily mixed in the same source file

what advantage would it have to mix together into the same source file the code for totally different targets with totally different parallelism models? I find it cleaner and simpler (apart from maybe trivial one-liner shaders that you can embed in a string) to keep them separated. Also makes debugging much easier.

> what advantage would it have to mix together into the same source file

It would be easier to share data structure declarations between the CPU and GPU side (MSL can already do this by sharing header files, a complete integration would just be the next step). GPU buffers and images could be described with [[attribute]] annotations (also very similar to what MSL does to extend C++ with shader semantics).

E.g. maybe a GPU buffer with vertex data could be described like this:

    [[gbu_buffer]] vertex_t vertices[] = {...};

...instead of calling a CreateBuffer(...) 3D-API functions, etc... (the code that's generated by the compiler might actually call a 'builtin' CreateBuffer function under the hood).

TL;DR: MSL is already halfway there by using standard C++ extended with custom attributes, it's just missing the last step to merge the Metal shader compiler with Apple's Clang toolchain.

PS: also if shader functions could be directly defined as 'regular functions' in CPU-side source code (and behind a special function pointer type), there would be no need for string literal shenanigans like this:

https://github.com/floooh/sokol-samples/blob/3f10c1a0620cec9...

(and shader compilation would happen during the regular build and would also generate regular compiler errors - with current toolchains that's only possible with a lot of build system magic)

Shaders are programs that run on the GPU. That term, like all others, come from the graphics processing background and history of GPUs.

Like the CPU, a GPU only understands its own instruction set. If you want to run software on a GPU, you must write it in a language that can compile to that instruction set. Making interpreters that run arbitrary languages doesn‘t really work on GPUs. They aren’t general purpose processors, after all.

Edit: researching GPU history might be helpful. In the old days, the era of early OpenGL and Direct3D, what you could do with a GPU was extremely limited. You could enable/disable some things and tweak some parameters, but what the GPU was really doing was out of your control. Google fixed function pipeline.

In more recent times, with Shaders and especially with the newer standards / APIs like Vulkan, Direct3D12, Metal and now WegGPU, the GPUs internals have been made much more flexible and blown open. We now have much more low level control. GPUs are quite complicated, so naturally working with these low level APIs is also complicated.

I hope this isn't too simplistic but it might help to think about what GPUs were originally designed for - banging out arrays of coloured dots at ridiculous speeds. Crudely (caveat: from my layman's understanding), when rendering graphics, a shader is a function whose input is a set of parameters that help identify where the pixel lies within its source triangle, and whose output is the appropriate colour for that pixel. A simple example might be to set the three vertices of a triangle respectively to red, green, and blue, and then interpolate the colour between those three points. When rendering the triangle, the shader is executed once for each pixel within it, and since each pixel's colour is independent of its neighbours', the problem is 'embarrassingly parallel' and can therefore be accelerated by running the same shader across multiple execution units varying only the input parameters.

Since computer graphics calculations are math-heavy, GPU cores are as you know excellent numerical processors, and therefore also great for other embarrassingly parallel maths problems; the challenge is that you have to think in broadly the same way as for graphical workloads about how to break up and execute the problem.

TL;DR; It has to work blazingly fast on all kinds of GPUs, which are not nearly as homogeneous as you might expect. The complex setup is the "lowest common denominator" of that ecosystem.

GPUs are stupendously good at ridiculously wide parallel processing, yet you still need to sequence these operations. Modern Graphics API setups like Vulkan and WebGPU are the currently best known way to write such systems. Shaders express the parallel algorithms, and the pipelines binds and synchronizes them together.

It's just legitimately the best way, as far as we know, to get the most bang out of your buck for that hardware while remaining vendor-agnostic. Anyways, a higher-level abstraction can always be built on top that provides you with the UX you desire, and we will see a bunch of them for WebGPU soon enough.

WebGPU is a low-level interface close to the hardware. If you don't care about having control over those details, it's not the right API to use. You'd have better luck with either a high-level wrapper or something else like WebGL.

WebGL still requires shader code I believe, and also uses graphic-specific terminology. My understanding is that there is no high-level native API for compute-specific use-cases. You are currently forced to use an API intended for graphics and then create abstracts for compute yourself.

I believe GPU.js exists to fill this void and facilitate simple GPU compute on the web without the need for all of the graphic-specific terminology and boiler plate code. But why a 3rd party library is needed for this makes little sense to me since a high-level native GPU (or hybrid) compute API seems like a great way to optimise many JS projects on the web today.

I must be missing something though because this seems like such a no brainer that if it was possible it would already be a thing.

Standard web APIs are a lot like kernel-level OS features. They have a ton of inertia to them, so they arguably should be limited to stuff that is infeasible in "userland". (There are additional reasons for that in the case of OSs, but that's the relevant one here).

If you had such a high-level API baked in, it would get leapfrogged by some library project within a few years, and then we'd be stuck with yet another API to support that no one actually uses.

Put briefly:

GPUs don't work like CPUs, at all, and need to massaged carefully to run efficiently. You can't just give a GPU an `int main()` and go ham. The APIs we get to use (DX12, Vulkan, Metal, WGPU) are all designed to abstract several GPU vendor's hardware architectures into an API that's just low-level enough to be efficient without forcing the app to explicitly target each GPU itself. There's no one way to run fast on everything with more than a trivial program.

As for why no 'universal high-level native GPU compute API' exists, GPU vendors don't want them because it's too much effort to implement. They'd much rather everything go through a low-level interface like Vulkan and the high level stuff get done by 'someone else' consuming Vulkan. Each new 'universal' API requires buy-in from 4+ different hardware vendors. Someone needs to painstaking specify the API. Each vendor must implement the API. Those implementations will be broken, just look at Android drivers, and now your 'simple' universal API is actually 4 different APIs that look almost the same except for strange esoteric behavior that appears on some hardware that is completely undocumented.

This is what OpenGL was and is why Vulkan, DX12 and Metal exist. The universal 'simple' API doesn't exist because it's too hard to make and not useful enough for the people programming GPUs regularly.

It’s not a dumb question but I think the issue is the abstraction layers you’re looking at.

Computing is hard in general. Your operating system and compilers do a lot of work that you don’t need to be aware of. In that way, your programming languages are high level abstractions and you trust the compiler to give you the best performance. However people still write simd intrinsics and assembly to get even more perf when the compiler can’t.

GPUs used to be much simpler , in that the abstracted a lot of the complexity away, letting the driver and hardware deal with things like a compiler would. But due to being so massively parallel, people wanted lower level control. This is almost exactly like people using simd intrinsics on CPUs.

So the programming model for GPUs became lower and lower level to allow for maximum performance. The higher level interface moved into frameworks and game engines.

Really, if you find this complex (and it is), you should start by learning a game engine. Just like dumping a first year student into simd would be hard, so is learning about GPUs from the graphics APIs.

Start high and go low, because a lot more will make sense as you understand the building blocks better from context.

It has to do with offloading computation to an external entity.

For example, lets take a simple example: CPU + DSP. I have an Atari Falcon here which sports something like that, but quite a few modern SoCs also have DSPs included (QCOM, TI, ...), or you have SBCs (single board computers, like Raspberry Pi) which sport an additional DSP chip on it.

Now what do you think the process will be to do the computations on the DSP?

You will need at the very minimum some way to hand data over to the DSP, execute some code, and retrieve whatever result back to the CPU. This can be as simple as just handing around memory pointers (in SoCs with "unified memory"), or as complicated as specifically triggering the communication line between these two chips (some fast serial connection, ... whatever). That also means you need buffers for that, i.e. the CPU has to somehow provide buffers to some API that deals with the handover; at the very minimum you need a buffer for the input data, the actual code, and for the output (unless you reuse the input). Now, you define some code to do whatever computation, who figures out the sizes of these buffers? For the code itself it can be easy, depending, but input and output?

From there it gets easily more complicated, and that is the very reason why there is so much "boilerplate" in APIs dealing with GPUs, and some drivers actually support OpenCL, but I'm not sure right now if that is made available in browsers even. This aims directly at "compute", but also brings all the aspects you need for properly offloading that with it. There is simply no way around that.

Loads of people have stated why easy GPU interfaces are difficult to create, but we solve many difficult things all the time.

Ultimately I think CPUs are just satisfactory for the vast vast majority of workloads. Servers rarely come with any GPUs to speak of. The ecosystem around GPUs is unattractive. CPUs have SIMD instructions that can help. There are so many reasons not to use GPUs. By the time anyone seriously considers using GPUs they're, in my imagination, typically seriously starved for performance, and looking to control as much of the execution details as possible. GPU programmers don't want an automagic solution.

So I think the demand for easy GPU interfaces is just very weak, and therefore no effort has taken off. The amount of work needed to make it as easy to use as CPUs is massive, and the only reason anyone would even attempt to take this on is to lock you in to expensive hardware (see CUDA).

For a practical suggestion, have you taken a look at https://arrayfire.com/ ? It can run on both CUDA and OpenCL, and it has C++, Rust and Python bindings.

A large part of the complexity comes from the GPU (often) having a separate memory from the CPU connected via a relatively slow bus. Another large part is that the GPU runs asynchronously from the CPU. Another large part is that you must write your code in a way that allows thousands of concurrent threads of execution or you won't run faster than the CPU. Another large part is the fact that the instruction set and features and general performance level of the GPU are unknown at development time and wildly different on common hardware. Despite all that, reaching near peak hardware utilization and real-time interactive performance is expected by users on all hardware regardless of the differences. All these things together mean that the obvious and simple designs for APIs unfortunately don't work very well.

I recently became aware of a high-level language called Futhark [1] that fits well in that space. It is statically-typed and functional.

Maybe some other array languages work in the GPU as well, I don't know.

[1] https://futhark-lang.org/

The short version is for me is, programming a GPU via a GPU API is like programming Assembly Language. If you want easy you use a higher level language like Python. Python is "usually" slower than assembly language but you can read a file, add up all the numbers in it, and print out the answer in 1 line of code where as in assembly here's lots of setup to read files, buffer their input, parse them into numbers, etc.

Maybe you want Use.GPU

https://usegpu.live/

I assume this type of API is not available natively because it's not how the interacting with the gpu works and is best left to libraries like gpu.js

I guess it is most likely you are not going to write a program multiplies two numbers, but you will need to multiply matrices passed in a certain way. those matrices are big ones, so you need to allocate some space for them, then load, then do the math, then use them in some other shader to paint some pixels on screen.

    GPU.exec(() => 2 * 2);

That's CUDA or SYCL (if you're willing to use C++).

It's not?

When writing with CUDA you also have a lot of boilerplate and things to think about like streams, blocks, grids. The list goes on and on.

To get to that level of abstraction you'd have to use something like pytorchs cuda backend.

It kind of is, because latest versions are increasingly supportive of standard C++.

The CUDA machinery is still there, just not exposed to work with directly.

Sure, if you use the compiler formerly now as Portland Group Compiler. nvc++ vs nvcc.

If you want high level abstractions use tools that provide those. But because GPUs are complicated and come with lots of caveats (see other comments) library authors need the comparatively low level access

You might be interested in https://futhark-lang.org

Reply on news.ycombinator.com