That's also how SunView worked on SunOS in 1982, and Sun's later GX graphics accelerated framebuffer driver worked in the 90's. The kernel managed the clipping list, and multiple processes shared the same memory, locking and respecting the clipping list and pixel buffers in shared memory (main memory, not GPU memory!), so multiple process could draw on different parts of the screen efficiently, without incurring system calls and context switches.

https://en.wikipedia.org/wiki/SunView

Programmers Reference Manual for the Sun Window System, rev C of 1 November 1983: Page 23, Locking and Clipping:

http://bitsavers.trailing-edge.com/pdf/sun/sunos/1.0/800-109...

But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.

So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)

More recently (on an archeological time scale, but for many years by now), MacOS, Windows, iOS, and Android have all developed ways of sharing graphics between multiple processes not only in shared main CPU memory, but also on the GPU, which greatly accelerates rendering, and is commonly used by web browsers, real time video playing and processing tools, desktop window managers, and user interface toolkits.

There are various APIs to pass handles to "External" or "IO Surface" shared GPU texture memory around between multiple processes. I've written about those APIs on Hacker News frequently over the years:

https://news.ycombinator.com/item?id=13534298

DonHopkins on Jan 31, 2017 | parent | context | favorite | on: Open-sourcing Chrome on iOS

It's my understanding that only embedded WKWebViews are allowed to enable the JIT compiler, but not UIWebViews (or in-process JavaScriptCore engines). WKWebView is an out-of-process web browser that uses IOSurface [1] to project the image into your embedding application and IPC to send messages.

So WKWebView's dynamically generated code is running safely firewalled in a separate address space controlled by Apple and not accessible to your app, while older UIWebViews run in the address space of your application, and aren't allowed to write to code pages, so their JIT compiler is disabled.

Since it's running in another process, WkWebView's JavaScriptEngine lacks the ability to expose your own Objective C classes to JavaScript so they can be called directly [2], but it does include a less efficient way of adding script message handlers that call back to Objective C code via IPC [3].

[1] https://developer.apple.com/reference/iosurface

[2] https://developer.apple.com/reference/javascriptcore/jsexpor...

[3] https://developer.apple.com/reference/webkit/wkusercontentco...

https://news.ycombinator.com/item?id=18763463

DonHopkins on Dec 26, 2018 | parent | context | favorite | on: WKWebView, an Electron alternative on macOS/iOS

Yes, it's a mixed bag with some things better and others worse. But having a great JavaScript engine with the JIT enabled is pretty important for many applications. But breaking up the browser into different processes and communicating via messages and sharing textures in GPU memory between processes (IOSurface, GL_TEXTURE_EXTERNAL_OES, etc) is the inextricable direction of progress, what all the browsers are doing now, and why for example Firefox had to make so many old single-process XP-COM xulrunner plug-ins obsolete.

IOSurface:

https://developer.apple.com/documentation/iosurface?language...

https://shapeof.com/archives/2017/12/moving_to_metal_episode...

GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

http://www.felixjones.co.uk/neo%20website/Android_View/

pcwalton on Dec 27, 2018 | prev [–]

Chrome and Firefox with WebRender are going the opposite direction and just putting all their rendering in the chrome process/"GPU process" to begin with.

DonHopkins on Dec 27, 2018 | parent [–]

Yes I know, that's exactly what I meant by "breaking up the browser into different processes". They used to all be in the same process. Now they're in different processes, and communicate via messages and shared GPU memory using platform specific APIs like IOSurface. So it's no longer possible to write an XP/COM plugin for the browser in C++, and call it from the renderer, because it's running in a different process, so you have to send messages and use shared memory instead. But then if the renderer crashes, the entire browser doesn't crash.

https://news.ycombinator.com/item?id=20313751

DonHopkins on June 29, 2019 | parent | context | favorite | on: Red Hat Expecting X.org to “Go into Hard Maintenan...

Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.

Mac OS/X and iOS IOSurface:

https://developer.apple.com/documentation/iosurface?language...

http://neugierig.org/software/chromium/notes/2010/08/mac-acc...

https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...

Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...

https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...

https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...

https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...

https://news.ycombinator.com/item?id=25997356

DonHopkins on Feb 2, 2021 | parent | context | favorite | on: VideoLAN is 20 years old today

>Probably do a multi-process media player, like Chrome is doing, with parsers and demuxers in a different process, and different ones for decoders and renderers. Knowing that you probably need to IPC several Gb/s between them. Chrome and other browsers and apps, and drivers like virtual webcams, and libraries like Syphon, can all pass "zero-copy" image buffers around between different processes by sharing buffers in GPU memory (or main memory too of course) and sending IPC messages pointing to the shared buffers.

That's how the browser's web renderer processes efficiently share the rendered images with the web browser user interface process, for example. And how virtual webcam drivers can work so efficiently, too.

Check out iOS/macOS's "IOSurface":

https://developer.apple.com/documentation/iosurface

>IOSurface Share hardware-accelerated buffer data (framebuffers and textures) across multiple processes. Manage image memory more efficiently.

>Overview: The IOSurface framework provides a framebuffer object suitable for sharing across process boundaries. It is commonly used to allow applications to move complex image decompression and draw logic into a separate process to enhance security.

And Android's "SurfaceTexture" and GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

>The image stream may come from either camera preview or video decode. A Surface created from a SurfaceTexture can be used as an output destination for the android.hardware.camera2, MediaCodec, MediaPlayer, and Allocation APIs. When updateTexImage() is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.

https://source.android.com/devices/graphics/arch-st

>The main benefit of external textures is their ability to render directly from BufferQueue data. SurfaceTexture instances set the consumer usage flags to GRALLOC_USAGE_HW_TEXTURE when it creates BufferQueue instances for external textures to ensure that the data in the buffer is recognizable by GLES.

And Syphon, which has a rich ecosystem of apps and tools and libraries:

http://syphon.v002.info

>Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.

Of course there's a VLC Syphon server:

https://github.com/rsodre/VLCSyphon