After 25 years, GTK joins the party …
With no native video drivers, moving a Vesa Haiku window with video playback still seems smoother than doing the same in Gnome/Kde under X11.
I mean.. shall we start a list of ways in which BeOS/Haiku have yet to "join the party" that the linux desktop has managed?
Juggling the various needs of one hell of a lot more users across a lot more platforms, with a lot more API-consuming apps to keep working on systems which are designed with a lot more component independence is a much harder problem to solve.
So the optimization only makes sense for a machine like yours with no native drivers. With any kind of GPU acceleration it will actually make things much slower. GTK doesn't do this because it would only be useful for that kind of machine running around 25 years ago.
Precisely one of the points of TFA is be able to use the "25 year old" hardware overlay support whenever possible (instead of the GPU) in order to save power, like Android (and classic Xv) does.
2. The provided code in Haiku doesn't appear to support overlays.
3. Yes, it's upsetting that a real API wasn't available for this until recently. But, it was of limited practical use without the entire display pipeline being moved to the GPU and without Wayland being established (X11 never had the API quite like this, classic XV is too limited to do what this is doing)
> BeOS and Haiku allowed exposure to kernel graphics buffers decades ago
In any case, X also allowed this for ages, with XV and XShm and the like. Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations and whatnot, and things went downhill since.
Enlightenments EFL was exclusively software for a long time, but is buttery smooth despite largely using redraw most of the time.
Hardware acceleration does not compensate for the lack of hard computer science knowledge about efficient memory operations, caching, and such
In the general case most effective hardware acceleration uses are supposed to increase latency.
Let's take another example for the analogy: you can run your LLM straight in the CPU and it will have awesome latency and very poor throughput; or you can take your sweet time to load the weights into your GPU/NPU/TPU/memory-in-compute device and run it hardware accelerated at lower energy cost and higher throughput.
No, XShm doesn't do that and the way XV does it is completely dependent on drivers. If you're using Glamour then XV won't use overlays at all. XShm uses a buffer created in CPU memory allocated by the client that the X server then has to copy to the screen.
> Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations
No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline. I'd like to see an actual benchmark for how a fullscreen 5k video with post-processing plays on Haiku without any hardware acceleration.
Fair enough. Even with XShmCreatePixmap, you are still never simply mmaping the card's actual entire framebuffer, unlike what BDirectWindow allows (if https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12.... is to believed, which is closer to something like DGA). In XShm, the server still has to copy your shared memory segment to the actual framebuffer.
(sorry for previous answer here, I misunderstood your comment)
> No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline.
Depends on what you mean by "post-processing". You can do many types of card-accelerated zero-copy post-processing using XV: colorspace conversion, scaling, etc. At the time, scaling the video in software or even just doing an extra memory copy per frame would have tanked frame rate -- Xine can be used to watch DVDs in PentiumII-level hardware. Obviously you cannot put the video in the faces of rotating 3D cubes, but this is precisely what I call "fancy animations".
There's a lot more than that. Please consider installing the latest version of VLC or something like that and checking all the available post-processing effects and filters. These aren't "fancy animations" and they're not rotating 3D cubes, they're basic features that a video player is required to support now. If you want to support arbitrary filters then you need to use the GPU. All these players stopped using XV ages ago, on X11 you'll get the GPU rendering pipeline too because of this.
I don't really see what's the point of making these condescending remarks like trying to suggest that everyone is stupid and is only interested in making wobbly windows and spinning cubes. Those have never been an actual feature of anything besides Compiz, which is a dead project.
Anyway, discussing about this is besides the point, and forgive me from the rant above.
If you really need GPU video filters then the GPU is obviously going to be the best way to implement them, there's no discussion possible about that. But the entire point of TFA is to (dynamically) go back to a model where the GPU is _not_ in the middle. And that model -- sans GPU -- happens to match what Xv was doing and is actually faster and less power consuming than to always blindly use the GPU which is where we are now post-Xv.
I don't know about anything else, but ffmpeg has some deinterlacing filters that run on the GPU. So your one example is a bad one.
>Anyway, discussing about this is besides the point, and forgive me from the rant above.
Next time can you please not post the rant? It's not interesting to parse through all that just to get to the point. It's also extremely uninteresting to have this conversation like "well I haven't personally used that so it must not be important". VLC and ffmpeg are used by millions (billions?) of people, so can we at least agree that neither of our own very particular and personal use cases are that important?
>But the entire point of TFA is to (dynamically) go back to a model where the GPU is _not_ in the middle.
No? Overlays are entirely driven by the GPU. The entire reason these are performant is because it's zero-copy from a GPU buffer to an overlay.
>And that model -- sans GPU -- happens to match what Xv was doing and is actually faster and less power consuming than to always blindly use the GPU which is where we are now post-Xv.
I have no idea what you're talking about. XV (with a driver that supports it) uses the GPU to do the overlays. If you aren't using any filters, this should have the same power consumption as XV.
The problem comes from people remembering all of the positive traits (often just promises) of a technology but either forgetting all the problems it had or the technology never being given enough of a chance for people to discover all the areas in which it is a bit crap.
BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!
This is also a big phenomenon in e.g. aviation. The greatest fighter jet ever is always the one that (tragically!) got cancelled before anyone could discover its weaknesses.
You can run Haiku today, so it's hardly unfalsifiable, nor an effect of nostalgia or whatever way you want to phrase it.
> BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!
"Even at that time" is just false. multi-user safe OSes abound in the 90s?
Excellent, so let's falsify it: how come 20 years later the best thing people really have to say about BeOS/Haiku is that is has smooth window dragging?
> multi-user safe OSes abound in the 90s?
Windows NT.
Linux and at least two other free unixes.
Countless proprietary unixes.
VMS.
The widespread desktop OSs in the 90s were not considered serious OSs even then, more accidents of backward-compatibility needs.
Over two decades later they don't have a 1.0 release.
People have been working damned hard to build things like Gtk, Qt, etc. not to mention Wayland, etc. all the while maintaining compatibility etc and I personally am happy for their efforts.
BeOS/HaikuOS is a product of a mid-90s engineering scene that predates the proliferation of GPUs, the web, and the set of programming languages that we work with today. There's nothing wrong with it in that context, but it's also not "better." Just different compromises.
The other one I see nostalgia nerds reach for is the Amiga. A system highly coupled to a set of custom chips that only made sense when RAM was as fast (or faster) than the CPU, whose OS had no memory protection, and which was easily outstripped in technical abilities by the early 90s by PCs with commodity ISA cards, etc. because of the development of economics of scale in computer manufacturing, etc. It was ahead of its time for about 2-3 years in the mid 80s, but in a way that was a dead end.
Anyways, what we have right now is a messy set of compromises. It doesn't hurt to go looking for simplifications, but it does hurt to pretend that the compromises don't exist for a reason.
EDIT: I would add though that "multiuser" as part of a desktop (or especially mobile) OS has maybe proven to be a pointless thing. The vast majority of Linux machines out there in the world are run in a single user fashion, even if they are capable of multiuser. Android phones, Chromebooks, desktop machines, and even many servers -- mostly run with just one master user. And we've also seen how rather not-good the Unix account permissions model is in terms of security, and how weak its timesharing of resources etc is in context of today's needs -- hence the development of cgroups, containers, and virtual machine / hypervisor etc.
Anybody who is serious about securing a single physical machine for multiuser access isn't doing it through multiple OS accounts, and is slicing it up by VMs, instead.
I do have a home (Windows) computer that gets used by multiple family members through accounts, but I think this isn't a common real world use case.
Money, marketing, politics, wanting to be part of the crowd, usually play a bigger role.
Drawing directly from app to kernel graphics buffers (or hw backed surfaces) and participating in composition are orthogonal I think. The compositor may be compositing the kernel or hw backed surfaces.
The compositor is the entire display server in Wayland, and is also the component responsible for bypassing composition when possible.
In a full-screen case AFAIK it's possible to skip the compositing step with X11, and maybe with Wayland too.
"Zero copy" seems a bit ambiguous term in graphics because there's the kind of copying where whole screen or window sized buffers are being copied about, and then there are compositing operations where various kinds of surfaces are inputs to the final rendering, where also pixels are copied around possibly several times in shaders but there aren't necessary extra texture sized intermediate buffers involved.
This is the trivial optimization all Wayland compositors do.
The neater trick is to do this for non-fullscreen content - and even just parts of windows - using overlay planes. Some compositors have their own logic for this, but libliftoff aims to generalize it.
Zero-copy is not really ambiguous, but to clarify: Wayland protocols are designed to maximize the cases where a buffer rendered by a client can be presented directly by the display hardware as-is (scanned out), without any intermediate operations on the content.
Note "maximize" - the content must be compatible with hardware capabilities. Wayland provides hints to stay within capabilities, but a client might pick a render buffer modifier/format that cannot be scanned out by the display hardware. GPUs have a lot of random limitations.
https://en.wikipedia.org/wiki/SunView
Programmers Reference Manual for the Sun Window System, rev C of 1 November 1983: Page 23, Locking and Clipping:
http://bitsavers.trailing-edge.com/pdf/sun/sunos/1.0/800-109...
But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
More recently (on an archeological time scale, but for many years by now), MacOS, Windows, iOS, and Android have all developed ways of sharing graphics between multiple processes not only in shared main CPU memory, but also on the GPU, which greatly accelerates rendering, and is commonly used by web browsers, real time video playing and processing tools, desktop window managers, and user interface toolkits.
There are various APIs to pass handles to "External" or "IO Surface" shared GPU texture memory around between multiple processes. I've written about those APIs on Hacker News frequently over the years:
https://news.ycombinator.com/item?id=13534298
DonHopkins on Jan 31, 2017 | parent | context | favorite | on: Open-sourcing Chrome on iOS
It's my understanding that only embedded WKWebViews are allowed to enable the JIT compiler, but not UIWebViews (or in-process JavaScriptCore engines). WKWebView is an out-of-process web browser that uses IOSurface [1] to project the image into your embedding application and IPC to send messages.
So WKWebView's dynamically generated code is running safely firewalled in a separate address space controlled by Apple and not accessible to your app, while older UIWebViews run in the address space of your application, and aren't allowed to write to code pages, so their JIT compiler is disabled.
Since it's running in another process, WkWebView's JavaScriptEngine lacks the ability to expose your own Objective C classes to JavaScript so they can be called directly [2], but it does include a less efficient way of adding script message handlers that call back to Objective C code via IPC [3].
[1] https://developer.apple.com/reference/iosurface
[2] https://developer.apple.com/reference/javascriptcore/jsexpor...
[3] https://developer.apple.com/reference/webkit/wkusercontentco...
https://news.ycombinator.com/item?id=18763463
DonHopkins on Dec 26, 2018 | parent | context | favorite | on: WKWebView, an Electron alternative on macOS/iOS
Yes, it's a mixed bag with some things better and others worse. But having a great JavaScript engine with the JIT enabled is pretty important for many applications. But breaking up the browser into different processes and communicating via messages and sharing textures in GPU memory between processes (IOSurface, GL_TEXTURE_EXTERNAL_OES, etc) is the inextricable direction of progress, what all the browsers are doing now, and why for example Firefox had to make so many old single-process XP-COM xulrunner plug-ins obsolete.
IOSurface:
https://developer.apple.com/documentation/iosurface?language...
https://shapeof.com/archives/2017/12/moving_to_metal_episode...
GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
http://www.felixjones.co.uk/neo%20website/Android_View/
pcwalton on Dec 27, 2018 | prev [–]
Chrome and Firefox with WebRender are going the opposite direction and just putting all their rendering in the chrome process/"GPU process" to begin with.
DonHopkins on Dec 27, 2018 | parent [–]
Yes I know, that's exactly what I meant by "breaking up the browser into different processes". They used to all be in the same process. Now they're in different processes, and communicate via messages and shared GPU memory using platform specific APIs like IOSurface. So it's no longer possible to write an XP/COM plugin for the browser in C++, and call it from the renderer, because it's running in a different process, so you have to send messages and use shared memory instead. But then if the renderer crashes, the entire browser doesn't crash.
https://news.ycombinator.com/item?id=20313751
DonHopkins on June 29, 2019 | parent | context | favorite | on: Red Hat Expecting X.org to “Go into Hard Maintenan...
Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.
Mac OS/X and iOS IOSurface:
https://developer.apple.com/documentation/iosurface?language...
http://neugierig.org/software/chromium/notes/2010/08/mac-acc...
https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...
Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...
https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...
https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...
https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
https://news.ycombinator.com/item?id=25997356
DonHopkins on Feb 2, 2021 | parent | context | favorite | on: VideoLAN is 20 years old today
>Probably do a multi-process media player, like Chrome is doing, with parsers and demuxers in a different process, and different ones for decoders and renderers. Knowing that you probably need to IPC several Gb/s between them. Chrome and other browsers and apps, and drivers like virtual webcams, and libraries like Syphon, can all pass "zero-copy" image buffers around between different processes by sharing buffers in GPU memory (or main memory too of course) and sending IPC messages pointing to the shared buffers.
That's how the browser's web renderer processes efficiently share the rendered images with the web browser user interface process, for example. And how virtual webcam drivers can work so efficiently, too.
Check out iOS/macOS's "IOSurface":
https://developer.apple.com/documentation/iosurface
>IOSurface Share hardware-accelerated buffer data (framebuffers and textures) across multiple processes. Manage image memory more efficiently.
>Overview: The IOSurface framework provides a framebuffer object suitable for sharing across process boundaries. It is commonly used to allow applications to move complex image decompression and draw logic into a separate process to enhance security.
And Android's "SurfaceTexture" and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
>The image stream may come from either camera preview or video decode. A Surface created from a SurfaceTexture can be used as an output destination for the android.hardware.camera2, MediaCodec, MediaPlayer, and Allocation APIs. When updateTexImage() is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.
https://source.android.com/devices/graphics/arch-st
>The main benefit of external textures is their ability to render directly from BufferQueue data. SurfaceTexture instances set the consumer usage flags to GRALLOC_USAGE_HW_TEXTURE when it creates BufferQueue instances for external textures to ensure that the data in the buffer is recognizable by GLES.
And Syphon, which has a rich ecosystem of apps and tools and libraries:
>Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.
Of course there's a VLC Syphon server:
This seems very strange to me. It’s how things would work with wl_shm, which is the baseline pixel-pushing interface in Wayland, but AFAIU Gtk uses EGL / Mesa, which in turn uses Linux dmabufs, which is how you do hardware-accelerated rendering / DRI on Linux today in general.
However, how precisely Linux dmabufs work in a DRI context is not clear to me, because the documentation is lacking, to say the least. It seems that you can ask map to dmabufs into memory, and you can create EGLSurfaces from them, but are they always mapped into CPU memory (if only kernel-side), or can they be bare GPU memory handles until the user asks to map them?
I’d hope for the latter, and if so, the only thing the work discussed in the article avoids is extra GPU-side blits (video decoding buffer to window buffer to screen), which is non-negligible but not necessarily the end of the world.
I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general.
ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is:
https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0...
The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)
Also, I wrote a significant part of GTK's current OpenGL renderer.
> But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.
This is absolutely and completely incorrect. Once we get things into GL, the texture is backed by a DMABUF on Linux. You never read it back into main memory. That would be very, very, very stupid.
> But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
Yes, the compositor is running in the GPU too. So of course we just tell the compositor what the GL texture id is and it composites if it cannot map that texture (again, because it's really a DMABUF) as a toplevel plane for hardware scanout without using 3d capabilities at all.
That doesn't mean unaccelerrated. It means it doesn't power up the 3d part of the GPU. It's the fastest way in/out with the least power. You can avoid "compositing" from a compositor too when things are done right.
> So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
Again, completely wrong.
> Check out iOS/macOS's "IOSurface":
Fun fact, I wrote the macos backend for GTK too. And yes, it uses IOSurface just like DMABUF works on Linux.
Could you please share your opinion on the toolkit, and its relation to others? Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those — what would you say, was it successful? Or is there still some legacy decisions that harm the project somewhat?
GTK 3 itself was trying to lose the tech debt of 2.x (which in turn 1.x). But they were still all wrapping a fundamentally crap API of X11 for graphics in this century.
GTK 4 changed that, and it now wraps a Wayland model of API. That drastically simplified GDK, which is why I could write a macOS backend in a couple of weeks.
It also completely changed how we draw. We no longer do immediate mode style (in the form of Cairo) and instead do a retained mode of draw commands. That allows for lots of new things you just couldn't do before with the old drawing model. It will also allow us to do a lot more fun things in the future (like threaded/tiled renderers).
The APIs all over the place were simplified and focused. I can't imagine writing an application the size of GNOME Builder again with anything less than GTK 4.
Hell, last GNOME cycle I rewrote Sysprof from scratch in a couple months, and it's become my co-pilot every day.
I'm sorry, I misinterpreted the paragraph in the article saying "exports" as meaning that it exports the pixels from GPU memory to CPU memory, not just passing a reference like GL_TEXTURE_EXTERNAL_OES and IOSurface does.
>GTK has already been using dmabufs since 4.0: When composing a frame, GTK translates all the render nodes (typically several for each widget) into GL commands, sends those to the GPU, and mesa then exports the resulting texture as a dmabuf and attaches it to our Wayland surface.
Perhaps I'd have been less confused if it said "passes a reference handle to the resulting texture in GPU memory" instead of "exports the resulting texture", because "exports" sounds expensive to me.
Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver, or build on top of GL_TEXTURE_EXTERNAL_OES, or is GL_TEXTURE_EXTERNAL_OES/SurfaceTexture just an Android or OpenGL ES thing that's an alternative to dmabufs in Linux? Do they work without any dependencies on X or Wayland or OpenGL, I hope? (Since pytorch doesn't use OpenGL.)
https://source.android.com/docs/core/graphics/arch-st
One practical non-gui use case I have for passing references to GPU textures between processes on Linux is pytorch: I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references to the resulting frames into another process (or even two -- each frame of video needs to be run through two different vision models) in another docker container running pytorch, sharing and multitasking the same GPU, possibly sending handles through a shared local file system or ipc (like how IOSurface uses Mach messages to magically send handles, or using unix domain sockets or ZeroMQ or something like that), but I don't know if it's something that's supported at the Linux operating system level (ubuntu), or if I'd have to drop down to the NVidia driver level to do that.
NVidia has some nice GPU video decompressor libraries, but they don't necessarily play well with pytorch in the same process, so I'd like to run them (or possibly ffmpeg) in a different process, but on the same GPU. Is it even possible, or am I barking up the wrong tree?
It would be ideal if ffmpeg had a built-in "headless" way to perform accelerated video decompression and push out GPU texture handles to other processes somehow, instead of rendering itself or writing pixels to files or touching CPU memory.
They are independent of the graphics subsystem altogether (although that is where they got their start, afaik). Your webcam also uses DMABUF. So if you want to display your webcam from a GTK 4 application, this GtkGraphicsOffload will help you take that DMABUF from your camera (which may not be mappable on CPU memory, but can DMA pass to your GPU), and display it in a GTK application. It could either be composited on the GPU, or mapped directly to scanout if the right conditions are met.
I wrote a library recently (libmks) and found the culprits in Qemu/VirGL/virtio_gpu that were preventing passing a DMABUF from inside a guest VM to the host. That stuff is all fixed now so theoretically you could even have a webcam in a VM which then uses a GTK 4 application to render with VirGL and the compositor submit the scene to the host OS which itself can set the planes correctly to get the same performance as if it were in the host OS.
> I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references
If you want this stuff with NVidia, and you're a customer, I highly suggest you tell your NVidia representative this. Getting them to use DMABUF in a fashion that can be used from other sub-systems would be fantastic.
But at it's core, if you were using Mesa and open drivers for some particular piece of hardware, yes it's capable of working given the right conditions.
It's not the same thing as what's being done here via Wayland. BeOS is more YOLO style direct access of the framebuffer contents, without solving any of the hard problems (I don't think it was really possible to do properly using the available hardware at the time).