GTK: Introducing Graphics Offload
blog.gtk.org
blog.gtk.org
I think it’s good that different graphics systems exist but it feels unnecessary that every team has to make their own discoveries how to handle the existing pieces.
If work were coordinated at least on the requirements and architecture level then I think a lot of synergies could be achieved. After that everyone can implement the architecture the way that works best for their use case, but on some common elements could be relied on.
Subsurfaces didn't have bug-free implementations for a while, so maybe some people avoided them. But I know some of us emulator programmers have been using them for output (especially because they can update asynchronously from the parent surface), and I think a couple media players do, too. It's not something that most applications really need.
The play button they show seems to be a good one, though. It is really nice to have it overlaid on the video.
Or instead of a triangular Play button, you could draw a big funny nose in some position and orientation, and the game would be to pause the video on a frame with somebody's face in it, with the nose in just the right spot.
I don't know why the vlc project is ignoring my prs.
What am I missing?
Interface Hall of Shame - QuickTime 4.0 Player (1999):
I might write some UI for mplayer/mpv on Motif some day, is not rocket science, you basically talk to both with sockets and sending commands. Kinda like mpc/mpd.
This then got driven into the ground with the "Fisher-Price GUI" that is the norm on mobile because you can't do anything with precision since you don't have a mouse.
I would actually really like to see a UI with just rectangles. Really. It's okay, designers. Take a deep breath and say: "GUIs aren't bound by the physical". BeOS and MacOS used to be very rectangular. Give us a nice nostaligia wave of fad design with rectangles, please.
Animations and drop shadows are another thing I'd like to see disappear.
https://www.folklore.org/StoryView.py?story=Round_Rects_Are_...
Or, take a look at the interaction mechanisms and palettes in MacPaint: https://www.punchkick.com/blog/2015/08/07/how-apple-has-shap...
Even in the 1990s, it was a lot of squareness.
And, as I point out, physical objects have rounded corners for a reason. Computers GUIs do not share that reason.
Rounded corners are a design choice. A different design choice can be made.
One other thing that people forget is that rounded things generally suck for packing.
I knew a guy who poked his eye out on a Windows 3.1 window corner. Super brutal, was running on only 16-colors too. It was the upper left corner, so there wasn't even a drop shadow to help.
But, from time to time, I sometimes use emwm with xfile and a bunch of light Motif apps with some Solaris9-themed GTK2/3 ones when there's no Motif alternative. Usable, with contrast and every menu it's sticky, so good enough for a netbook.
Couldn’t the rounded corners of a video also be an overlay?
I’m sure I’m missing something here, but the article does not explain that point.
No because the clipping is done in the client after the content is drawn. The client doesn't have the full screen contents. To make it work with an overlay, the clipping would have to be moved to the server. There could be another extension that lets you pass an alpha mask texture to the server to use as a clip mask. But this doesn't exist (yet?)
That means you have to power up the 3d part of the GPU to do that (because the renderer does it in shaders).
Where as if you add some 9 pixels of black above/below to account for the rounded corner, there is no clipping of the video and you can use hardware scanout planes.
That's important because keeping the 3d part of the GPU turned off is a huge power savings. And the scanout plane can already scale for you to the correct size.
Once support for hardware planes becomes more common in Wayland compositors, this can be tied to ultimately allow no-copy rendering to the display for non-fullscreen applications, which for video playback (incl. likes of Youtube) equals to reduced CPU & GPU usage and less power draw, as well as reduced latency.
What.
The original design of X actually encouraged a separate surface / Window for each single widget on your UI. This was actually removed in Gtk+3 ("windowless widgets"). And now they are bringing it back just for wayland ("subsurfaces"). As far as I can read, it is practically the same concept.
GTK3 got rid of windowed widgets because Keith Packard introduced the Xrender extension, which basically added 2D compositing to X, which was the last remaining use for subwindows for every widget.
The excuse that was used when introducing windowless widgets is to reduce tearing/noise during resizing, as Gtk+ had trouble synchronizing the resizing of all the windows at the same time.
Yes, that's the point. When you can tell Xrender to efficiently composite some pixmaps then there's really no reason to use sub-windows ever.
>You make your toolkit's programmer's life more complicated, not less, by having windowless widgets (at the very minimum you now have to complicate your code with offsets and clip regions and the like).
No, you still had to have offsets and clip regions before too because the client still had to set and update those. And it was more complicated because when you made a sub-window every single bit of state like that had to be synchronized with the X server and repeatedly copied over the wire. With client-side rendering everything is simply stored in the client and never has to deal with that problem.
There is, or we would not be having subsurfaces on Wayland or this entire discussion in the first place.
Are you seriously arguing that the only reason to using windows in Xorg is to have composition? People were using Xshape/Xmisc and the like to handle the lack of alpha channels in the core protocol? This is not what I remember. I would be surprised if Xshape even worked on non-top level windows. heck, even MOTIF had windowless widgets (called gadgets iirc), and the purpose most definitely was not composition-related.
No. The subsurfaces in Wayland are only designed for two things:
1. Direct scan-out, as in TFA. (Because a subsurface can be directly translated to a dmabuf)
2. Embedding content from one toolkit/library into another. (Because without it, lots of glue code would be needed)
It's discouraged to use them otherwise, they would complicate things for no benefit.
>Are you seriously arguing that the only reason to using windows in Xorg is to have composition?
If you mean sub-windows, yes, that and consequently because of the way that XRDB worked. I don't see why you would ever use them just for input events within the same toolkit, they don't do anything special there.
https://www.donhopkins.com/home/catalog/unix-haters/x-window...
That's what happened when I simply resized stock xcalc several times in a row to different sizes and aspect ratios. I wonder if they've finally figured out how to fix that bug in 35 years?
That was why I was compelled to hack an X11 window manager to take a command line argument telling it which window id to treat as the root window, then ran xcalc, discovered its window id with "xwininfo" or some such utility, then ran the window manager on the calculator, putting window frames around each of the calculator's buttons, so you could resize them, move them around, open and close them to icons, etc! That was a truly customizable calculator.
Sure it lets you do fast 2d acceleration but we don't use 2d accel infrastructure anywhere anymore.
Subsurfaces have been in Wayland since the beginning of the protocol.
This is simply getting them to work on demand so we can do something X (or Xv) could never do since Drawables would get moved to new memory (that may not even be mappable on the CPU-side) on every frame.
And that's to actually use the scanout plane correctly to avoid powering up the 3d part of the GPU when doing video playback on composited systems.
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/t...
"This global is a factory interface, allowing clients to inform which type of presentation the content of their surfaces is suitable for."
Note that "global" refers to the interface, not the setting. Which Wayland compositor has the equivalent feature of "xrandr --output [name] --set TearFree off"?
Here you are saying that latency of 250ms is unnoticeable, which is utter nonsense.
Yeah, that is utter nonsense. Just try it for yourself, instead of pulling statements like that out of thin air. You would notice a difference in tens of ms even when editing text. Why do you think people cannot stand writing with VSCode for example?
Re: moving/evolving shapes, I did not think I had to clarify that the brain is a massively parallel system with multiple modes of operation. Editing text does not require you to reprocess all visual signals from scratch, because that is not how the visual cortex works. The perceived latency when editing text is between pressing a key and your brain telling you "my eyes have detected a change on the screen. I will assume that it is the result of me pressing a key". It does NOT take 250ms to make this type of assumption, and is basically how our vision operates. It's a prediction engine, not a CCD sensor.
Which people? Every recent study I've seen shows VSCode as the most popular code editor by a large margin. Maybe latency isn't as important as you think?
>Are you saying that latency in the order of 250ms when editing text is unnoticeable?
No. Sorry for the info dump here but I'm going to make it absolutely clear so there's no confusion. The latency of the entire system is the latency of the human operator plus the latency of the computer. My statement is that, assuming you have a magical computer that computes frames and displays pixels faster than the speed of light, the absolute minimum bound of this system for the average person is 250ms. You only see lower response time averages in extreme situations like with pro athletes: so basically, not computer programmers who actually spend much more time thinking about problems, and going to meetings, than they actually spend typing.
Now let's go back to reality: with a standard 60Hz monitor, the theoretical latency added by display synchronization is a maximum of about 16.67ms. That's the theoretical MAXIMUM assuming the software is fully optimized and performs rendering as fast as possible, and your OS has realtime guarantees so it doesn't preempt the rendering thread, and the display hardware doesn't add any latency. So at most, you could reduce the total system latency by about 6% just by optimizing the software. You can't go any higher than that.
However, none of those things are true in practice. Making the renderer use damage tracking everywhere significantly complicates the code and may not even be usable in some situations like syntax highlighting where the entire document state may need to be recomputed after typing a single character. All PC operating systems may have significant unpredictable lag caused by the driver. All display hardware using a scanline-based protocol also still has significant vblank periods. Adding these up you may be able to sometimes get a measurement of around 1ms of savings by doing things this way, in exchange for massively complicating your renderer, and with a high standard deviation. Meaning that you likely will perceive the total latency as being HIGHER because of all the stuttering. This is less than 1% of the total latency in the system and it's not even going to be consistent or perceptible.
Now instead consider you've got a 360Hz monitor. The theoretical maximum you can save here is about 2.78ms. This can give you a CONSISTENT 5% latency reduction against the old monitor as long as the software can keep up with it. Optimizing your software for this improves it in every other situation too, versus the other solution which could make it worse. If it doesn't make it worse, it could only save another theoretical 1% and ONLY in a badly perceptible way. It just doesn't make sense to optimize for this less than 1% when it's mostly just caused by the hardware limitations and nobody actually cares about it and they're happy to use VSCode anyway without all this.
So again, you can avoid these accusations of "utter nonsense" when it's clear you're arguing against something that I never said.
>The perceived latency when editing text is between pressing a key and your brain telling you "my eyes have detected a change on the screen.
Your brain needs to actually process what was typed. Prediction isn't helping you type at all, if it did then the latency wouldn't matter anyway. If you're not just writing boilerplate code then you may have to stop to think many many times while you're coding too.
Only if the DE or application can't keep up. Oh, it overran the frame time, so let's just dump half of it to screen...
I'm still baffled that this is a problem. I tried ray tracing a hierarchy of rectangles decades ago and it was pretty fast. Should be better than real time today, and that's doing it the expensive way. I suppose the challenge is composting stuff from different processes every frame.
> I suppose the challenge is composting stuff from different processes every frame.
This is a significant part of the issue, though the fact that synchronization essentially means waiting is also another one (without synchronizing you get partial frames but those partial frames are still feedback for the user that makes the desktop feel more responsive).
And you're not limited to two images. As the frame rate increases, the number of images increases and the tear becomes less noticeable. Blur Busters explains:
https://blurbusters.com/faq/benefits-of-frame-rate-above-ref...
When touch typing, fingers work decoupled from the eyes anyway, unless you are waiting for the intellisense or copilot prompt, that is usually constrained by language servers anyway, not the framerate.
Most humans will actually prefer the "slightly faster" option. (Obviously if you can do both, then they'd prefer that; but given the trade-off...)
For example, with a 60 Hz display and vsync, game actions might be shown up to 16 ms later than without vsync, which is ages in FPS.
I've seen no game consoles that allow you to turn vsync off, because it would be awful. No idea why this placebo persists in PC gaming.
how about Graphics Offload?
Also only two of the four changes mentioned in the mail are about CVEs.
Yes, you're technically right that this would have been possible years ago but it wasn't actually ever done, because X11 never had the ability to do it at the same time as using compositing.
You really need to add context to these statements, because _right now_ I am using through Xorg a program which uses a frigging colormap, which is as non-RGB as it gets. The entire reason Xlib has this "WhitePixel" and XGetPixel and XYPixmap and other useless functions which normally fetch a lot of ire is because it tries to go out of its way to support practically other-wordly color visuals and image formats. If anything, I'd say it is precisely RGB which has the most problems with X11, specially when you go more than 24bpp.
> there for attaching GPU buffers
None of this is about the GPU, but about about directly presenting images for _hardware_ composition using direct scan-out, hardware layers or not. Exactly what Xv is about, and the reason Xv supports formats like YUV.
> There isn't any incentive to implement this in X11 either because X11 is supposed to work over the network and none of this stuff would
As if that prevented any of the extensions done to X11 in the last three decades, including Xv.
That doesn't change what I said. The colormap is mapping indexes to RGB palette entries and technically doesn't even support any other color spaces. Nothing about that is "non-RGB". The other visuals are just various other ways to do RGB. If this isn't making sense to you, think about how this would be implemented in the driver.
>None of this is about the GPU, but about about directly presenting images for _hardware_ composition using direct scan-out
I'm sorry? What do you suppose is doing the direct scan-out to the monitor if not the GPU?
>As if that prevented any of the extensions done to X11 in the last three decades, including Xv.
That's not relevant, XV actually does support running over the network as long as you don't use the SHM functions. But regardless, yes, it actually did. The main example being indirect GLX which hasn't been updated in decades because it's not feasible to run it over the network anymore.
You need to dynamically change stacking of subsurfaces on a per-frame basis when doing the CRTC.
I agree though it would require a lot of changes to the server and no one is in the mood (like, dynamically decide whether I composite this window or push it to a Xv port or hardware plane? practically inconceivable in the current graphics stack, albeit it is not a technical X limitation per-se). This entire feature is also going to be pretty pointless in Wayland desktop space either way because no one is in the mood either -- your dmabufs are going to end up in the GPU anyway for the foreseeable future, just because of the complexity of liftoff, variability of GPUs, and the like.
You'll need API to remap the Drawable to the scanout plane from a compositor on a per-frame basis (so when submitting the CRTC) and the compositor isn't in control of the CRTC. So...
Nothing in Xorg is a minor change because it has to be tested with every window manager and compositor before you can even think about merging.
> At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
> What are the limitations?
> At the moment, graphics offload will only work with Wayland on Linux. There is some hope that we may be able to implement similar things on MacOS, but for now, this is Wayland-only. It also depends on the content being in dmabufs.
When GNOME adopted GTK as its foundation, there was a clear separation between GTK and the GNOME libraries, back in the 1.0 - 2.0 days.
Eventually GNOME needs became GTK roadmap.
The rest one can find on the history books.
Just like you, I love to write articles about user interface stuff all the time, too. Just in the past week:
My enthusiastic but balanced response to somebody who EMPHATICALLY DEMANDED PIE MENUS ONLY for GIMP, and who loves pie fly, but pushed my button by defending the name GIMP by insisting that instead of the GIMP project simply and finally conceding its name is offensive, that our entire society adapt by globally re-signifying a widely known offensive hurtful word (so I suggested he first go try re-signifying the n-word first, and see how that went):
https://news.ycombinator.com/item?id=38233793
(While I would give more weight to the claim that the name GIMP is actually all about re-signifying an offensive term if it came from a qualified and empathic and wheelchair using interface designer like Haraldur Ingi Þorleifsson, I doubt that’s actually the real reason, just like it’s not white people’s job to re-signify the n-word by saying it all the time...)
Meet the man who is making Iceland wheelchair accessible one ramp at a time:
https://scoop.upworthy.com/meet-the-man-who-is-making-icelan...
Elon Musk apologises after mocking laid-off Twitter worker, questioning his disability:
https://www.abc.net.au/news/2023-03-08/elon-musk-haraldur-th...
The article about redesigning GIMP we were discussing credited Blender with being the first to show what mouse buttons do what at the bottom of the screen, which actually the Lisp Machine deserves credit for, as far as I know:
https://news.ycombinator.com/item?id=38237231
I made a joke about how telling GIMP developers to make it more like Photoshop was like telling RMS to develop Open Software for Linux, instead of Free Software for GNU/Linux, and somebody took the bait so I flamed about the GIMP developer’s lack of listening skills:
https://news.ycombinator.com/item?id=38238274
Somebody used the phrase "Easy as pie” in a discussion about user interface design so I had to chime in:
https://news.ycombinator.com/item?id=38239113
Discussion about HTML Web Components, in which I confess my secret affair with XML, XSLT, obsolete proprietary Microsoft technologies, and Punkemon pie menus:
https://news.ycombinator.com/item?id=38253752
Deep interesting discussion about Blender 4.0 release notes, focusing on its historic development and its developer’s humility and openness to its users’ suggestions, in which I commented on its excellent Python integration.
https://news.ycombinator.com/item?id=38263171
Comment on how Blender earned its loads of money and support by being responsive to its users.
https://news.ycombinator.com/item?id=38232404
Dark puns about user interface toolkits and a cheap shot at Motif, with an analogy between GIMP and Blender:
https://news.ycombinator.com/item?id=38263088
A content warning to a parent who wanted to know which videos their 8-year-old should watch on YouTube to learn Blender:
https://news.ycombinator.com/item?id=38288629
Posing with a cement garden gnome flipping the bird with Chris Toshok and Miguel de Icaza and his mom at GDC2010:
https://www.facebook.com/photo/?fbid=299606531754&set=a.5173...
https://www.facebook.com/photo/?fbid=299606491754&set=a.5173...
https://www.facebook.com/photo/?fbid=299606436754&set=a.5173...
You'll have to read the rest of the comment and follow the links to know what it says, because I already wrote and summarized it, and don't want to write it again just for you, because I don't believe you'd read it a second time if you didn't read it the first time. Just use ChatGPT, dude.
Then you will see that it has a lot to do with GTK and GNOME and GIMP, even including exclusive photos of Miguel de Icaza and his mom with a garden gnome flipping the bird.
https://donhopkins.medium.com/the-x-windows-disaster-128d398...
>The Motif Self-Abuse Kit
>X gave Unix vendors something they had professed to want for years: a standard that allowed programs built for different computers to interoperate. But it didn’t give them enough. X gave programmers a way to display windows and pixels, but it didn’t speak to buttons, menus, scroll bars, or any of the other necessary elements of a graphical user interface. Programmers invented their own. Soon the Unix community had six or so different interface standards. A bunch of people who hadn’t written 10 lines of code in as many years set up shop in a brick building in Cambridge, Massachusetts, that was the former home of a failed computer company and came up with a “solution:” the Open Software Foundation’s Motif.
>What Motif does is make Unix slow. Real slow. A stated design goal of Motif was to give the X Window System the window management capabilities of HP’s circa-1988 window manager and the visual elegance of Microsoft Windows. We kid you not.
>Recipe for disaster: start with the Microsoft Windows metaphor, which was designed and hand coded in assembler. Build something on top of three or four layers of X to look like Windows. Call it “Motif.” Now put two 486 boxes side by side, one running Windows and one running Unix/Motif. Watch one crawl. Watch it wither. Watch it drop faster than the putsch in Russia. Motif can’t compete with the Macintosh OS or with DOS/Windows as a delivery platform.
Also, you get XFT and UTF-8 support thru fontconfig and XFT on every Motif based software. Far from the propietary Motif of 1996...
Compare it to Gnome and dconf...
And neither of those things are even toolkits like GTK: "GIMP Tolkit" is an image editor, and "GNOME Tolkit" is a desktop environment.
And even if you ignore the two typos and the two mis-namings, his whole point is factually incorrect, and jdub was correct and not a revisionist when he said "That is ahistorical, and the misnaming doesn't help make your point".
I'm simply giving him the benefit of the doubt, and asking him to explain what he means, or why not only his main point was wrong, but also why he got both of the names wrong two times in a row, and thinks an image editor and a desktop environment are incorrectly spelled toolkits.
And he still hasn't explained, or admitted he made two typos and misnamed two projects in a row while trying to make an incorrect point, while claiming to be an expert tech writer, and accusing someone who was correct of being a revisionist, so the jury is still out. But your theory it's clearly a typo just doesn't wash. Maybe they're the names of his own forks, or maybe he's just a charlatan, who knows? ;) Why don't you ask him yourself.
Exactly? If you're still holding out for GTK to be a non-Linux toolkit in 2023 then you're either an incredibly misguided contributor and/or ignorant of the history behind the toolkit. The old GTK does not exist anymore, you either use GNOME's stack or you don't.
As you say, the old GTK is dead. GNOME murdered it. I mourn it.
They've made a whole lot of objective and subjective missteps in the past, but I don't think it's fair to characterize them as an evil party here. They did the work, they reap the rewards, and they take the flak for the myriad of different ways the project could/should have gone.
It's not that they're the evil party, it's just that they should stop feeling so sorry for themselves that so few people want to use their image editor because of its terrible user interface caused by the fact that they refuse to listen to their users, and it has a terribly offensive name that they refuse to change.
At least they still have a fanatical following of MAGA incel edgelords and ESR sycophants who love it BECAUSE it has an offensive name, so they still have that hard core fanbase to appeal to.
They're as self-sabotaging as RMS himself, and they don't deserve to play the victim or to have a pity party, especially when they try to throw it for themselves.
Now that's an ahistorical conspiracy theory. Those diverse desktop environments contributed hugely to GTK, GNOME just didn't use their work or consider it helpful unless it directly related to their desktop... and of course none of that work will relate to their desktop. Nobody is going to fully "kiss the ring" unless they get something out of it, and even back in the GTK3 days it was plainly clear that GNOME didn't care about you if you didn't care about GNOME.
Now, GNOME's "coup" or "killing" of GTK is completely fine by Open Source standards. Even encouraged. I don't stand against the concepts of what they're doing, but they could have done a lot better than fighting third-parties tooth-and-nail. GNOME should be a proud project that leads the GNU movement, and instead it was reduced to a bunch of squabbling supremacists that made their userbase an adversary. I say all that as someone who quite likes modern GTK and writes apps in it.
No? Where exactly do you think I've theorized about the existence of a conspiracy? Because I've actually said the exact opposite: there isn't a conspiracy and no one is cooperating at all. There's no evil group of developers secretly planning to sabotage everything. It's just the usual bad communication and planning that happens with a distributed team.
>Those diverse desktop environments contributed hugely to GTK, GNOME just didn't use their work
Can you name what any of these contributions were? Because I've never seen them. I've seen contributions here and there, lots of minor bug fixes, but nothing major.
>Nobody is going to fully "kiss the ring" unless they get something out of it
Avoid this rhetoric please. These open source projects are a volunteer collaboration. No one's kissing any rings or trying to get something out of the maintainers, other than the usual: everyone helps each other write and maintain the code.
>but they could have done a lot better than fighting third-parties tooth-and-nail. GNOME should be a proud project that leads the GNU movement
I really don't know what you're talking about here, but disagreeing about technical things isn't "fighting tooth-and-nail". That's a normal part of any project.
Personally I don't think anyone should care about leading the GNU movement, that's been plagued by petty infighting and drama since the very beginning.
My memory of the early days is consistent with what the wikipedia page says about GIMP. It was cross-platform on the typical Unix workstations that were around the UC Berkeley campus labs and XCF. This was things like Solaris, SunOS, HP-UX, Ultrix, and Irix.
Students in this milieu were just as likely to have some BSD variant on their home PC as Linux. I think it was later during and after the "Beowulf" scientific computing period when Linux started to dominate as the Unix-like platform for open source development.
I don't really have time for that though, I only wrote the macOS port because I had some extra holiday hacking time.
A bit of reading if you are interested:
https://learn.microsoft.com/en-us/windows/win32/direct3darti...
Maybe that is because full-screen direct scanout doesn't take much (if anything) in a toolkit, it's almost purely a compositor feature.
> The idea is that you don’t have to copy lots of pixel data around, and instead just pass a file descriptor between kernel subsystems.
So like sendfile for graphics?
https://www.man7.org/linux/man-pages/man2/sendfile.2.html
It's a pretty awesome system call. We should have more of those.
I believe neither Xv nor GL-based renderers do that, even before we discuss hw accel.
The older consoles like the NES had a pixel processing unit that generated the picture, you need to emulate it's state and interaction with the rest of the system, possibly cycle-by-cycle, which makes it not possible to do on the GPU as a shader or whatever.
This is kind of a niche use case for sure but it's interesting.
I guess it would allow the user/system to customize/tweak the compositing process somehow, but for what purpose?
I would guess lower latency since the compositor is running the show trying to maintain full frame rate. The processes still have to cooperate somehow.
I think I found a minor typo:
GTK 4.14 will introduce a GtkGraphicsOffload widget, whose only job it is to give a hint that GTK should try to offload the content of its child widget by attaching it to a subsurface instead of letting GSK process it like it usually does.
I think that "GSK" nesr the end should just be "GTK". It's not a very near miss on standard qwerty, though ...
Back when videos wouldn't show up in screen shots and their position on screen could sometimes gets out of sync with the window they were playing in?
I never fully understood what was happening but my theory at the time was that the video was being sent to the video card separate from the rendered windows.
(Sad it is written in c++)
But what did surprise me even more: I was expecting GTK, and moreover the 4, to be on par with valve software.
On linux, I would not event think to code a modern system hardware accelerated GFX component which is not drm(dmabuf)/vulkan.
Unless the article is obsolete itself?
Not to mention GL is a massive and gigantic kludge/bloat compared to vulkan (mostly due to the glsl compiler). So this is good to let it go.
GL is only deprecated on Mac from what I understand.
I humbly don't think I am the right guy for that: I use raw x11, and will use raw wayland once the steam client is finally and actually wayland enabled, even though I may do the jump before... because "valve": their "pressure-vessel" is buggy, and I am helping fixing alsa support in SDL3, all that is starting... namely if I code something GFX related that would be more a wayland compositor on drm(dmabufs)/vulkan, probably based on valve one, either in plain and simple C (compiling with tcc, cproc/qbe, simple-cc/qbe), minimal to 0 dependencies without the latest ISO tantrums, or x86_64 assembly (before a port to riscv in the far future).
But never say never.
>It is legacy and started to be retired.
The standard itself, but the implementations are still being maintained and new extensions are being added. It is still a solid base to build upon.
After 25 years, GTK joins the party …
With no native video drivers, moving a Vesa Haiku window with video playback still seems smoother than doing the same in Gnome/Kde under X11.
The problem comes from people remembering all of the positive traits (often just promises) of a technology but either forgetting all the problems it had or the technology never being given enough of a chance for people to discover all the areas in which it is a bit crap.
BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!
This is also a big phenomenon in e.g. aviation. The greatest fighter jet ever is always the one that (tragically!) got cancelled before anyone could discover its weaknesses.
You can run Haiku today, so it's hardly unfalsifiable, nor an effect of nostalgia or whatever way you want to phrase it.
> BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!
"Even at that time" is just false. multi-user safe OSes abound in the 90s?
Excellent, so let's falsify it: how come 20 years later the best thing people really have to say about BeOS/Haiku is that is has smooth window dragging?
> multi-user safe OSes abound in the 90s?
Windows NT.
Linux and at least two other free unixes.
Countless proprietary unixes.
VMS.
The widespread desktop OSs in the 90s were not considered serious OSs even then, more accidents of backward-compatibility needs.
Over two decades later they don't have a 1.0 release.
People have been working damned hard to build things like Gtk, Qt, etc. not to mention Wayland, etc. all the while maintaining compatibility etc and I personally am happy for their efforts.
BeOS/HaikuOS is a product of a mid-90s engineering scene that predates the proliferation of GPUs, the web, and the set of programming languages that we work with today. There's nothing wrong with it in that context, but it's also not "better." Just different compromises.
The other one I see nostalgia nerds reach for is the Amiga. A system highly coupled to a set of custom chips that only made sense when RAM was as fast (or faster) than the CPU, whose OS had no memory protection, and which was easily outstripped in technical abilities by the early 90s by PCs with commodity ISA cards, etc. because of the development of economics of scale in computer manufacturing, etc. It was ahead of its time for about 2-3 years in the mid 80s, but in a way that was a dead end.
Anyways, what we have right now is a messy set of compromises. It doesn't hurt to go looking for simplifications, but it does hurt to pretend that the compromises don't exist for a reason.
EDIT: I would add though that "multiuser" as part of a desktop (or especially mobile) OS has maybe proven to be a pointless thing. The vast majority of Linux machines out there in the world are run in a single user fashion, even if they are capable of multiuser. Android phones, Chromebooks, desktop machines, and even many servers -- mostly run with just one master user. And we've also seen how rather not-good the Unix account permissions model is in terms of security, and how weak its timesharing of resources etc is in context of today's needs -- hence the development of cgroups, containers, and virtual machine / hypervisor etc.
Anybody who is serious about securing a single physical machine for multiuser access isn't doing it through multiple OS accounts, and is slicing it up by VMs, instead.
I do have a home (Windows) computer that gets used by multiple family members through accounts, but I think this isn't a common real world use case.
Money, marketing, politics, wanting to be part of the crowd, usually play a bigger role.
Drawing directly from app to kernel graphics buffers (or hw backed surfaces) and participating in composition are orthogonal I think. The compositor may be compositing the kernel or hw backed surfaces.
The compositor is the entire display server in Wayland, and is also the component responsible for bypassing composition when possible.
In a full-screen case AFAIK it's possible to skip the compositing step with X11, and maybe with Wayland too.
"Zero copy" seems a bit ambiguous term in graphics because there's the kind of copying where whole screen or window sized buffers are being copied about, and then there are compositing operations where various kinds of surfaces are inputs to the final rendering, where also pixels are copied around possibly several times in shaders but there aren't necessary extra texture sized intermediate buffers involved.
This is the trivial optimization all Wayland compositors do.
The neater trick is to do this for non-fullscreen content - and even just parts of windows - using overlay planes. Some compositors have their own logic for this, but libliftoff aims to generalize it.
Zero-copy is not really ambiguous, but to clarify: Wayland protocols are designed to maximize the cases where a buffer rendered by a client can be presented directly by the display hardware as-is (scanned out), without any intermediate operations on the content.
Note "maximize" - the content must be compatible with hardware capabilities. Wayland provides hints to stay within capabilities, but a client might pick a render buffer modifier/format that cannot be scanned out by the display hardware. GPUs have a lot of random limitations.
> BeOS and Haiku allowed exposure to kernel graphics buffers decades ago
In any case, X also allowed this for ages, with XV and XShm and the like. Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations and whatnot, and things went downhill since.
Enlightenments EFL was exclusively software for a long time, but is buttery smooth despite largely using redraw most of the time.
Hardware acceleration does not compensate for the lack of hard computer science knowledge about efficient memory operations, caching, and such
In the general case most effective hardware acceleration uses are supposed to increase latency.
Let's take another example for the analogy: you can run your LLM straight in the CPU and it will have awesome latency and very poor throughput; or you can take your sweet time to load the weights into your GPU/NPU/TPU/memory-in-compute device and run it hardware accelerated at lower energy cost and higher throughput.
No, XShm doesn't do that and the way XV does it is completely dependent on drivers. If you're using Glamour then XV won't use overlays at all. XShm uses a buffer created in CPU memory allocated by the client that the X server then has to copy to the screen.
> Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations
No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline. I'd like to see an actual benchmark for how a fullscreen 5k video with post-processing plays on Haiku without any hardware acceleration.
Fair enough. Even with XShmCreatePixmap, you are still never simply mmaping the card's actual entire framebuffer, unlike what BDirectWindow allows (if https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12.... is to believed, which is closer to something like DGA). In XShm, the server still has to copy your shared memory segment to the actual framebuffer.
(sorry for previous answer here, I misunderstood your comment)
> No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline.
Depends on what you mean by "post-processing". You can do many types of card-accelerated zero-copy post-processing using XV: colorspace conversion, scaling, etc. At the time, scaling the video in software or even just doing an extra memory copy per frame would have tanked frame rate -- Xine can be used to watch DVDs in PentiumII-level hardware. Obviously you cannot put the video in the faces of rotating 3D cubes, but this is precisely what I call "fancy animations".
There's a lot more than that. Please consider installing the latest version of VLC or something like that and checking all the available post-processing effects and filters. These aren't "fancy animations" and they're not rotating 3D cubes, they're basic features that a video player is required to support now. If you want to support arbitrary filters then you need to use the GPU. All these players stopped using XV ages ago, on X11 you'll get the GPU rendering pipeline too because of this.
I don't really see what's the point of making these condescending remarks like trying to suggest that everyone is stupid and is only interested in making wobbly windows and spinning cubes. Those have never been an actual feature of anything besides Compiz, which is a dead project.
Anyway, discussing about this is besides the point, and forgive me from the rant above.
If you really need GPU video filters then the GPU is obviously going to be the best way to implement them, there's no discussion possible about that. But the entire point of TFA is to (dynamically) go back to a model where the GPU is _not_ in the middle. And that model -- sans GPU -- happens to match what Xv was doing and is actually faster and less power consuming than to always blindly use the GPU which is where we are now post-Xv.
I don't know about anything else, but ffmpeg has some deinterlacing filters that run on the GPU. So your one example is a bad one.
>Anyway, discussing about this is besides the point, and forgive me from the rant above.
Next time can you please not post the rant? It's not interesting to parse through all that just to get to the point. It's also extremely uninteresting to have this conversation like "well I haven't personally used that so it must not be important". VLC and ffmpeg are used by millions (billions?) of people, so can we at least agree that neither of our own very particular and personal use cases are that important?
>But the entire point of TFA is to (dynamically) go back to a model where the GPU is _not_ in the middle.
No? Overlays are entirely driven by the GPU. The entire reason these are performant is because it's zero-copy from a GPU buffer to an overlay.
>And that model -- sans GPU -- happens to match what Xv was doing and is actually faster and less power consuming than to always blindly use the GPU which is where we are now post-Xv.
I have no idea what you're talking about. XV (with a driver that supports it) uses the GPU to do the overlays. If you aren't using any filters, this should have the same power consumption as XV.
https://en.wikipedia.org/wiki/SunView
Programmers Reference Manual for the Sun Window System, rev C of 1 November 1983: Page 23, Locking and Clipping:
http://bitsavers.trailing-edge.com/pdf/sun/sunos/1.0/800-109...
But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
More recently (on an archeological time scale, but for many years by now), MacOS, Windows, iOS, and Android have all developed ways of sharing graphics between multiple processes not only in shared main CPU memory, but also on the GPU, which greatly accelerates rendering, and is commonly used by web browsers, real time video playing and processing tools, desktop window managers, and user interface toolkits.
There are various APIs to pass handles to "External" or "IO Surface" shared GPU texture memory around between multiple processes. I've written about those APIs on Hacker News frequently over the years:
https://news.ycombinator.com/item?id=13534298
DonHopkins on Jan 31, 2017 | parent | context | favorite | on: Open-sourcing Chrome on iOS
It's my understanding that only embedded WKWebViews are allowed to enable the JIT compiler, but not UIWebViews (or in-process JavaScriptCore engines). WKWebView is an out-of-process web browser that uses IOSurface [1] to project the image into your embedding application and IPC to send messages.
So WKWebView's dynamically generated code is running safely firewalled in a separate address space controlled by Apple and not accessible to your app, while older UIWebViews run in the address space of your application, and aren't allowed to write to code pages, so their JIT compiler is disabled.
Since it's running in another process, WkWebView's JavaScriptEngine lacks the ability to expose your own Objective C classes to JavaScript so they can be called directly [2], but it does include a less efficient way of adding script message handlers that call back to Objective C code via IPC [3].
[1] https://developer.apple.com/reference/iosurface
[2] https://developer.apple.com/reference/javascriptcore/jsexpor...
[3] https://developer.apple.com/reference/webkit/wkusercontentco...
https://news.ycombinator.com/item?id=18763463
DonHopkins on Dec 26, 2018 | parent | context | favorite | on: WKWebView, an Electron alternative on macOS/iOS
Yes, it's a mixed bag with some things better and others worse. But having a great JavaScript engine with the JIT enabled is pretty important for many applications. But breaking up the browser into different processes and communicating via messages and sharing textures in GPU memory between processes (IOSurface, GL_TEXTURE_EXTERNAL_OES, etc) is the inextricable direction of progress, what all the browsers are doing now, and why for example Firefox had to make so many old single-process XP-COM xulrunner plug-ins obsolete.
IOSurface:
https://developer.apple.com/documentation/iosurface?language...
https://shapeof.com/archives/2017/12/moving_to_metal_episode...
GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
http://www.felixjones.co.uk/neo%20website/Android_View/
pcwalton on Dec 27, 2018 | prev [–]
Chrome and Firefox with WebRender are going the opposite direction and just putting all their rendering in the chrome process/"GPU process" to begin with.
DonHopkins on Dec 27, 2018 | parent [–]
Yes I know, that's exactly what I meant by "breaking up the browser into different processes". They used to all be in the same process. Now they're in different processes, and communicate via messages and shared GPU memory using platform specific APIs like IOSurface. So it's no longer possible to write an XP/COM plugin for the browser in C++, and call it from the renderer, because it's running in a different process, so you have to send messages and use shared memory instead. But then if the renderer crashes, the entire browser doesn't crash.
https://news.ycombinator.com/item?id=20313751
DonHopkins on June 29, 2019 | parent | context | favorite | on: Red Hat Expecting X.org to “Go into Hard Maintenan...
Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.
Mac OS/X and iOS IOSurface:
https://developer.apple.com/documentation/iosurface?language...
http://neugierig.org/software/chromium/notes/2010/08/mac-acc...
https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...
Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...
https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...
https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...
https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
https://news.ycombinator.com/item?id=25997356
DonHopkins on Feb 2, 2021 | parent | context | favorite | on: VideoLAN is 20 years old today
>Probably do a multi-process media player, like Chrome is doing, with parsers and demuxers in a different process, and different ones for decoders and renderers. Knowing that you probably need to IPC several Gb/s between them. Chrome and other browsers and apps, and drivers like virtual webcams, and libraries like Syphon, can all pass "zero-copy" image buffers around between different processes by sharing buffers in GPU memory (or main memory too of course) and sending IPC messages pointing to the shared buffers.
That's how the browser's web renderer processes efficiently share the rendered images with the web browser user interface process, for example. And how virtual webcam drivers can work so efficiently, too.
Check out iOS/macOS's "IOSurface":
https://developer.apple.com/documentation/iosurface
>IOSurface Share hardware-accelerated buffer data (framebuffers and textures) across multiple processes. Manage image memory more efficiently.
>Overview: The IOSurface framework provides a framebuffer object suitable for sharing across process boundaries. It is commonly used to allow applications to move complex image decompression and draw logic into a separate process to enhance security.
And Android's "SurfaceTexture" and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
>The image stream may come from either camera preview or video decode. A Surface created from a SurfaceTexture can be used as an output destination for the android.hardware.camera2, MediaCodec, MediaPlayer, and Allocation APIs. When updateTexImage() is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.
https://source.android.com/devices/graphics/arch-st
>The main benefit of external textures is their ability to render directly from BufferQueue data. SurfaceTexture instances set the consumer usage flags to GRALLOC_USAGE_HW_TEXTURE when it creates BufferQueue instances for external textures to ensure that the data in the buffer is recognizable by GLES.
And Syphon, which has a rich ecosystem of apps and tools and libraries:
>Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.
Of course there's a VLC Syphon server:
This seems very strange to me. It’s how things would work with wl_shm, which is the baseline pixel-pushing interface in Wayland, but AFAIU Gtk uses EGL / Mesa, which in turn uses Linux dmabufs, which is how you do hardware-accelerated rendering / DRI on Linux today in general.
However, how precisely Linux dmabufs work in a DRI context is not clear to me, because the documentation is lacking, to say the least. It seems that you can ask map to dmabufs into memory, and you can create EGLSurfaces from them, but are they always mapped into CPU memory (if only kernel-side), or can they be bare GPU memory handles until the user asks to map them?
I’d hope for the latter, and if so, the only thing the work discussed in the article avoids is extra GPU-side blits (video decoding buffer to window buffer to screen), which is non-negligible but not necessarily the end of the world.
I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general.
ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is:
https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0...
The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)
Also, I wrote a significant part of GTK's current OpenGL renderer.
> But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.
This is absolutely and completely incorrect. Once we get things into GL, the texture is backed by a DMABUF on Linux. You never read it back into main memory. That would be very, very, very stupid.
> But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
Yes, the compositor is running in the GPU too. So of course we just tell the compositor what the GL texture id is and it composites if it cannot map that texture (again, because it's really a DMABUF) as a toplevel plane for hardware scanout without using 3d capabilities at all.
That doesn't mean unaccelerrated. It means it doesn't power up the 3d part of the GPU. It's the fastest way in/out with the least power. You can avoid "compositing" from a compositor too when things are done right.
> So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
Again, completely wrong.
> Check out iOS/macOS's "IOSurface":
Fun fact, I wrote the macos backend for GTK too. And yes, it uses IOSurface just like DMABUF works on Linux.
Could you please share your opinion on the toolkit, and its relation to others? Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those — what would you say, was it successful? Or is there still some legacy decisions that harm the project somewhat?
GTK 3 itself was trying to lose the tech debt of 2.x (which in turn 1.x). But they were still all wrapping a fundamentally crap API of X11 for graphics in this century.
GTK 4 changed that, and it now wraps a Wayland model of API. That drastically simplified GDK, which is why I could write a macOS backend in a couple of weeks.
It also completely changed how we draw. We no longer do immediate mode style (in the form of Cairo) and instead do a retained mode of draw commands. That allows for lots of new things you just couldn't do before with the old drawing model. It will also allow us to do a lot more fun things in the future (like threaded/tiled renderers).
The APIs all over the place were simplified and focused. I can't imagine writing an application the size of GNOME Builder again with anything less than GTK 4.
Hell, last GNOME cycle I rewrote Sysprof from scratch in a couple months, and it's become my co-pilot every day.
I'm sorry, I misinterpreted the paragraph in the article saying "exports" as meaning that it exports the pixels from GPU memory to CPU memory, not just passing a reference like GL_TEXTURE_EXTERNAL_OES and IOSurface does.
>GTK has already been using dmabufs since 4.0: When composing a frame, GTK translates all the render nodes (typically several for each widget) into GL commands, sends those to the GPU, and mesa then exports the resulting texture as a dmabuf and attaches it to our Wayland surface.
Perhaps I'd have been less confused if it said "passes a reference handle to the resulting texture in GPU memory" instead of "exports the resulting texture", because "exports" sounds expensive to me.
Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver, or build on top of GL_TEXTURE_EXTERNAL_OES, or is GL_TEXTURE_EXTERNAL_OES/SurfaceTexture just an Android or OpenGL ES thing that's an alternative to dmabufs in Linux? Do they work without any dependencies on X or Wayland or OpenGL, I hope? (Since pytorch doesn't use OpenGL.)
https://source.android.com/docs/core/graphics/arch-st
One practical non-gui use case I have for passing references to GPU textures between processes on Linux is pytorch: I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references to the resulting frames into another process (or even two -- each frame of video needs to be run through two different vision models) in another docker container running pytorch, sharing and multitasking the same GPU, possibly sending handles through a shared local file system or ipc (like how IOSurface uses Mach messages to magically send handles, or using unix domain sockets or ZeroMQ or something like that), but I don't know if it's something that's supported at the Linux operating system level (ubuntu), or if I'd have to drop down to the NVidia driver level to do that.
NVidia has some nice GPU video decompressor libraries, but they don't necessarily play well with pytorch in the same process, so I'd like to run them (or possibly ffmpeg) in a different process, but on the same GPU. Is it even possible, or am I barking up the wrong tree?
It would be ideal if ffmpeg had a built-in "headless" way to perform accelerated video decompression and push out GPU texture handles to other processes somehow, instead of rendering itself or writing pixels to files or touching CPU memory.
They are independent of the graphics subsystem altogether (although that is where they got their start, afaik). Your webcam also uses DMABUF. So if you want to display your webcam from a GTK 4 application, this GtkGraphicsOffload will help you take that DMABUF from your camera (which may not be mappable on CPU memory, but can DMA pass to your GPU), and display it in a GTK application. It could either be composited on the GPU, or mapped directly to scanout if the right conditions are met.
I wrote a library recently (libmks) and found the culprits in Qemu/VirGL/virtio_gpu that were preventing passing a DMABUF from inside a guest VM to the host. That stuff is all fixed now so theoretically you could even have a webcam in a VM which then uses a GTK 4 application to render with VirGL and the compositor submit the scene to the host OS which itself can set the planes correctly to get the same performance as if it were in the host OS.
> I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references
If you want this stuff with NVidia, and you're a customer, I highly suggest you tell your NVidia representative this. Getting them to use DMABUF in a fashion that can be used from other sub-systems would be fantastic.
But at it's core, if you were using Mesa and open drivers for some particular piece of hardware, yes it's capable of working given the right conditions.
So the optimization only makes sense for a machine like yours with no native drivers. With any kind of GPU acceleration it will actually make things much slower. GTK doesn't do this because it would only be useful for that kind of machine running around 25 years ago.
Precisely one of the points of TFA is be able to use the "25 year old" hardware overlay support whenever possible (instead of the GPU) in order to save power, like Android (and classic Xv) does.
2. The provided code in Haiku doesn't appear to support overlays.
3. Yes, it's upsetting that a real API wasn't available for this until recently. But, it was of limited practical use without the entire display pipeline being moved to the GPU and without Wayland being established (X11 never had the API quite like this, classic XV is too limited to do what this is doing)
I mean.. shall we start a list of ways in which BeOS/Haiku have yet to "join the party" that the linux desktop has managed?
Juggling the various needs of one hell of a lot more users across a lot more platforms, with a lot more API-consuming apps to keep working on systems which are designed with a lot more component independence is a much harder problem to solve.
It's not the same thing as what's being done here via Wayland. BeOS is more YOLO style direct access of the framebuffer contents, without solving any of the hard problems (I don't think it was really possible to do properly using the available hardware at the time).
Wayland for "security" is cargo culting smartphone user problems. It's not actually a real issue.
I use the keyboard/mouse sharing in X11 (via synergy) and I have for 20 years. It is vitally important to my workflow. It works on dozens of different OSes including linux. But not the waylands linuxes. Any graphical environment that can't do this is useless to me. Might as well not even release the waylands at all (see how silly applying my personal preferences globally seems?).
Here's 6 CVEs just from last month. Check the mailing lists and you'll see many of these going back for years and years.
https://lists.x.org/archives/xorg/2023-October/061506.html
https://lists.x.org/archives/xorg/2023-October/061514.html
And before you say this is not what you meant, the X server and X client libraries do very little anymore besides parsing untrusted input and passing it somewhere else. That's its main purpose and it's completely bad at it. And because it's X, this input can also come from over the network too so every normal memory bug can also be an RCE. This is probably the single biggest attack vector on a desktop system aside from the toolkit. It's the exact wrong thing for anyone to grant access to every input on the system.
This is not just my personal opinion or me giving anecdotes either, this is paraphrasing what I've heard X developers say after many years of patching these bugs. But that's not even the whole problem as I'll explain shortly.
>But for actual computers you control it just isn't (a problem). Wayland for "security" is cargo culting smartphone user problems. It's not actually a real issue.
Yes it is a problem and no it's not cargo culting. Practically speaking the X11 security model means every X client gets access to everything including all your passwords (and the root password) as you type them, and subsequently lets every X client spawn arbitrary root processes and get access to your whole filesystem including your private keys and insert kernel modules or do whatever. If you actually think this "isn't a real issue" then you should just stop using passwords, stop protecting your private keys, run every program as root, and disable memory protection: because that's what this actually means in practice. No I'm not exaggerating. The security model of X11 has no idea about client boundaries at all. This is completely unacceptable on any other OS but for some reason it's become a meme to say that only smartphones need to care about this. Really? Come on.
>I use the keyboard/mouse sharing in X11 (via synergy) and I have for 20 years. It is vitally important to my workflow. It works on dozens of different OSes including linux. But not the waylands linuxes. Any graphical environment that can't do this is useless to me.
X11 can't do it securely so I would say that's as useless as not implementing the feature, if you have to compromise your security in order to get it.
The feature will be implemented in Wayland eventually when the design for a secure API is finished. There are people working on it now. In comparison, X11 is probably never going to gain a secure way to do that.
It is cargo culting. It's not actually a problem that my applications are powerful and can do what I want them to do. It is a problem that other locked down OSes like Macs and smartphone systems are not in the user's control and programs cannot do many things by design. This is because on those systems the users are not in control of what is running and the OS makers believe they know better. If they can't do it it is useless (no qualification re: fantasy security issues needed).
... sharing keyboard/mouse with synergy/barrier/etc is secure.
This is not something the X maintainers can say. They can encourage people not to do it but if they stop maintaining that feature then the complaints start to roll in because someone somewhere was using it. If you think this situation is awful then yes, you're starting to get it: X is in a bad spot where these broken insecure features are holding else everything back and will continue to do so as long as people depend on it. At best they can disable it by default and make it hard to accidentally re-enable it, which is what they've already done.
>That's not something a normal desktop install does.
Yes, most normal desktop installs don't use X11 in any capacity. They use Microsoft Windows.
>It's not actually a problem that my applications are powerful and can do what I want them to do.
I notice you didn't actually respond to my comment about stopping using passwords and private keys and running everything as root. Because I'd bet even you draw a line somewhere, in a place where you think it's a risk to give an application too much power.
>It is a problem that other locked down OSes like Macs and smartphone systems are not in the user's control and programs cannot do many things by design.
This has absolutely nothing to do with Linux or even on those systems either. It's not actually a problem there. If you have root on the system then you are in control and can do whatever you want anyway. The purpose of setting security boundaries and not running everything as root is because not everything needs to access everything else all the time. The security model you're suggesting became obsolete by the mid 1990s.
And let me say this again so it's perfectly clear. When you use X11 there is effectively no security boundary between any X11 clients. So if you start up your root terminal or you use sudo or anything else like that, then any other X11 client on the system also gets root. This is unacceptable and I can't believe I still have to continually point this out to long time Linux users that should be technical enough to understand. It doesn't even matter if you personally think it's fine to run everything as root: maybe you do. But as a user you should have enough understanding of the system to know that this absolutely is not ok for lots of other users and it's simply not appropriate to be shipped as the default in the year 2023.
These are not fantasy issues, these are actual issues that the underlying system was purposely designed to fix. X11 pokes a huge gaping hole in it.
>sharing keyboard/mouse with synergy/barrier/etc is secure.
No. On a typical X11 install it's not, because it relies on insecure APIs.
You mean this one very small piece? That seems a bit hyperbolic
Probably because HTML is at the very top of the stack, while this is much lower... Without everything below it on the stack, HTML is just a text file.
All I meant was: if the API of HTML is simple, then why does GTK's API have to be that complicated.