Comment by audidude | Hacker News Reader

Comment by audidude | Hacker News Reader

Parent Full thread

I can't respond to everything incorrect in this, because it's way to long to read. But from the very start...

Also, I wrote a significant part of GTK's current OpenGL renderer.

> But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.

This is absolutely and completely incorrect. Once we get things into GL, the texture is backed by a DMABUF on Linux. You never read it back into main memory. That would be very, very, very stupid.

> But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.

Yes, the compositor is running in the GPU too. So of course we just tell the compositor what the GL texture id is and it composites if it cannot map that texture (again, because it's really a DMABUF) as a toplevel plane for hardware scanout without using 3d capabilities at all.

That doesn't mean unaccelerrated. It means it doesn't power up the 3d part of the GPU. It's the fastest way in/out with the least power. You can avoid "compositing" from a compositor too when things are done right.

> So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)

Again, completely wrong.

> Check out iOS/macOS's "IOSurface":

Fun fact, I wrote the macos backend for GTK too. And yes, it uses IOSurface just like DMABUF works on Linux.

I’m not the parent poster, I’m just tryin go to grab this opportunity that I “met” someone so familiar with GTK :)

Could you please share your opinion on the toolkit, and its relation to others? Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those — what would you say, was it successful? Or is there still some legacy decisions that harm the project somewhat?

> Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those

GTK 3 itself was trying to lose the tech debt of 2.x (which in turn 1.x). But they were still all wrapping a fundamentally crap API of X11 for graphics in this century.

GTK 4 changed that, and it now wraps a Wayland model of API. That drastically simplified GDK, which is why I could write a macOS backend in a couple of weeks.

It also completely changed how we draw. We no longer do immediate mode style (in the form of Cairo) and instead do a retained mode of draw commands. That allows for lots of new things you just couldn't do before with the old drawing model. It will also allow us to do a lot more fun things in the future (like threaded/tiled renderers).

The APIs all over the place were simplified and focused. I can't imagine writing an application the size of GNOME Builder again with anything less than GTK 4.

Hell, last GNOME cycle I rewrote Sysprof from scratch in a couple months, and it's become my co-pilot every day.

Thanks for the comment and for your work!

Thank you for the correction, it's a relief! That's nice work.

I'm sorry, I misinterpreted the paragraph in the article saying "exports" as meaning that it exports the pixels from GPU memory to CPU memory, not just passing a reference like GL_TEXTURE_EXTERNAL_OES and IOSurface does.

>GTK has already been using dmabufs since 4.0: When composing a frame, GTK translates all the render nodes (typically several for each widget) into GL commands, sends those to the GPU, and mesa then exports the resulting texture as a dmabuf and attaches it to our Wayland surface.

Perhaps I'd have been less confused if it said "passes a reference handle to the resulting texture in GPU memory" instead of "exports the resulting texture", because "exports" sounds expensive to me.

Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver, or build on top of GL_TEXTURE_EXTERNAL_OES, or is GL_TEXTURE_EXTERNAL_OES/SurfaceTexture just an Android or OpenGL ES thing that's an alternative to dmabufs in Linux? Do they work without any dependencies on X or Wayland or OpenGL, I hope? (Since pytorch doesn't use OpenGL.)

https://source.android.com/docs/core/graphics/arch-st

One practical non-gui use case I have for passing references to GPU textures between processes on Linux is pytorch: I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references to the resulting frames into another process (or even two -- each frame of video needs to be run through two different vision models) in another docker container running pytorch, sharing and multitasking the same GPU, possibly sending handles through a shared local file system or ipc (like how IOSurface uses Mach messages to magically send handles, or using unix domain sockets or ZeroMQ or something like that), but I don't know if it's something that's supported at the Linux operating system level (ubuntu), or if I'd have to drop down to the NVidia driver level to do that.

NVidia has some nice GPU video decompressor libraries, but they don't necessarily play well with pytorch in the same process, so I'd like to run them (or possibly ffmpeg) in a different process, but on the same GPU. Is it even possible, or am I barking up the wrong tree?

It would be ideal if ffmpeg had a built-in "headless" way to perform accelerated video decompression and push out GPU texture handles to other processes somehow, instead of rendering itself or writing pixels to files or touching CPU memory.

> Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver,

They are independent of the graphics subsystem altogether (although that is where they got their start, afaik). Your webcam also uses DMABUF. So if you want to display your webcam from a GTK 4 application, this GtkGraphicsOffload will help you take that DMABUF from your camera (which may not be mappable on CPU memory, but can DMA pass to your GPU), and display it in a GTK application. It could either be composited on the GPU, or mapped directly to scanout if the right conditions are met.

I wrote a library recently (libmks) and found the culprits in Qemu/VirGL/virtio_gpu that were preventing passing a DMABUF from inside a guest VM to the host. That stuff is all fixed now so theoretically you could even have a webcam in a VM which then uses a GTK 4 application to render with VirGL and the compositor submit the scene to the host OS which itself can set the planes correctly to get the same performance as if it were in the host OS.

> I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references

If you want this stuff with NVidia, and you're a customer, I highly suggest you tell your NVidia representative this. Getting them to use DMABUF in a fashion that can be used from other sub-systems would be fantastic.

But at it's core, if you were using Mesa and open drivers for some particular piece of hardware, yes it's capable of working given the right conditions.

Reply on news.ycombinator.com