Hacker News Reader

Comment by zokier | Hacker News Reader

While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute (/graphics), how are the limited GPU time and memory allocated between them? Can you set priorities and limits like you can with other resources?

I feel these sort of "operational" questions are often neglected in the discussions, but considering how GPU is increasingly being used in wide range of applications (both for graphics and compute) I think it's becoming relevant to think how they play together.

View on HN

The CUDA driver handles multitasking issues between processes, as each process has no knowledge of any other process.

But within your own application, yes, you can create multiple streams and assign each a priority.

If you are in Apple land, you can set the priority of your Metal command queues. Even with the integrated GPU, it is logically a separate device and you have limited control over it. And the comment about having its own compiler allows for some weird behavior. For example, it is possible to use an iPad to do Metal GPU programming using the Swift Playgrounds app: Use the standard boilerplate Swift code that sets an app up for executing a Metal shader. But instead of telling it to run a compiled kernel from the kernel library that would have been created during the application compile step on a desktop PC, you pass it Metal C++ source code as a plain old Swift string. The Metal runtime will compile and run it! You get no warnings about errors in your code; it either works or it doesn't, but alas, it shows that the compiler is indeed somewhere else :)

The driver handles time slicing between processes, mapping virtual memory to real memory, etc.

You're right that this is an actual consideration for programs. I don't know about CUDA/GPUGPU stuff, but for games, you need to manage residency of your resources, essentially telling the driver which resources (buffers/textures) are critical and can't be unloaded to make space for other apps, and which aren't critical.

Scheduling and allocation are done by the GPU driver, e.g. the CUDA runtime, with some hardware/firmware assistance from the GPU, which also contains one or more microcontrollers which may perform some of the tasks required for this.

Can you point me to reading material about how the CUDA runtime does this with hardware assistance? I looked but I have been unable to find any thing persuasive in this direction.

Reply on news.ycombinator.com