Hard numbers in the Wayland vs. X11 input latency discussion
mort.coffee
mort.coffee
Let me state up front that I have no idea why Wayland would have this additional latency. That said, having been active in the computer community at the 'birth' of X11 (yes I'm that old) I can tell you that there was, especially early on, a constant whine about screen latency. Whether it was cursor response or xterm scrolling. When "workstations" became a thing, they sometimes had explicit display hardware for just the mouse because that would cut out the latency of rendering the mouse in the frame. (not to mention the infamous XOR patent[1])
As a result of all this whinging, the code paths that were between keyboard/mouse input and their effect on the screen, were constantly being evaluated for ways to "speed them up and reduce latency." Wayland, being something relatively "new" compared to X11, has not had this level of scrutiny for as long. I'm looking forward to folks fixing it though.
A fullscreen app ought to be taking a fast path through / around the compositor.
I'll be reading a dream of spring in my grave at this rate.
I understand I'm complaining about free things, but this is a forced change for the worse for so long. Wayland adoption should have been predicated on a near universal superiority in all input and display requirements.
Intel and AMD and Nvidia and Arm makers should be all in on a viable desktop Linux as a consortium. Governments should be doing the same because a secure Linux desktop is actually possible. It is the fastest path to showcasing their CPUs and 3d bling, advanced vector /computer hardware.
Wayland simply came at a time to further the delay of the Linux desktop, at a time when Windows was attempting to kill Windows with its horrid tiles and Apple staunchly refused a half billion in extra market cap by offering osx on general x86.
There's rarely any such thing as "universal superiority", usually you're making a tradeoff. In the case of X vs Wayland it's usually latency vs. tearing. Personally I'm happy with Wayland because there was a time when watching certain videos with certain media players on Linux was incredibly painful because of how blatant and obtrusive the tearing was. Watching the same video under Wayland worked fine.
Early automobiles didn't have "universal superiority" to horses, but that wasn't an inhibitor to adoption.
-Problems with non western input systems
-Accessibility
-Remote control(took around 2 years to be stable I think?)
-Bad color management
Then there's the things that did work in x11 but not in wayland:
-Bad support for keymapping(the input library says keymapping should be implemented by the compositor, gnome says not in scope, so we have a regression)
-bad nvidia support for the first two years? three years?
While these things are compositor/hw vendor faults, the rush to use wayland and nearly every distro making it as default, forced major regressions and wayland kinda promised to improve the x11 based experience.
I get there was cruft in x. But the moment selected was a barrier to Linux desktop adoption precisely when the greatest opportunity in decades was present.
And the desktop was reimplemented.
Now in this period kde and gnome both decided to do rewrites, Ubuntu did their own desktop, got it up to snuff, and abandoned it. The lunacy wasn't just Wayland.
If we are complaining the gnome compositor sucks... I mean , should that be the goddamn reference implementation? What percent of desktops are gnome, 80% at least? If the gnome composting ready for primetime, then Wayland isn't ready for primetime.
I use Sway, which uses a different compos[i]tor than Gnome. I would like to see similar results for wlroots, Sway's compositor, though I'm not actually interested enough to do the experiment (I guess that would be comparing Sway with i3). Cursor lag in Sway is not enough to bother me. I have on occasion used Gnome on the same machine(s), and never been bothered by lag.
As others have pointed out, Wayland is a protocol, not a compositor.
But the Wayland protocol requires a compositor, so here we are.
Given that Gnome runs on both X and Wayland, it might be interesting to hear from the Gnome authors on the performance differences.
With regards to accessibility, what problems have you had exactly?
Fedora shipped a broken screen reader for 8 years.
Edit: Found it https://ar.al/2024/06/23/fedora-has-been-shipping-with-a-bro...
Not a bug, apparently. https://gitlab.freedesktop.org/libinput/libinput/-/issues/10...
That may be fine.
Neo2 does not work. Neo has 3 modifier keys, Gnome/Mutter/Wayland/Whatever does only support two. Neo2 has a compose key, Wayland does not honor it.
I use mod4 for navigation (arrow keys, site up) and compose for Slavish (read Polish) input (źżąę)
Which rush? It has been done by only a small fraction of distros like Fedora, after years of development of the first wayland compositors. Fedora main purpose has always been to implement bleeding edge tech stuff early so that bugs get found and fixed before people using more stable distros have to suffer from it.
Nobody has been forced in any regression and x11 has continued to be available until now and there is no sign that the most conservative distros will drop x11 support anytime soon.
Not that I mind particularly, X is fine and everything works as expected. I can even have the integrated (amd gpu) take care of the desktop while the Nvidia gpu does machine learning (I need all the vram I can get, the desktop alone would be 100-150mb of vram) - then I start a game on steam and the nvidia gpu gets used.
Funnily enough I had wayland enabled by default when I installed the system and I didn't understand why I was getting random freeze and artifacts for weeks. Then I realized I was not using X11
I feel like at some point this is a cop-out. Wayland is a protocol but its also a "system" involving many components. If the product as a whole doesn't work well then its still a failure regardless of which component's fault it is.
Its a little like responding to someone saying we haven't reached the year of linux on the desktop by saying: well actually linux is just the kernel and its been ready for the desktop for ages. Technically true but also missing the point.
> multi-window X applications
Then a client is going to query the server and ask request to do stuff via a unix socket. In fact, you don't even need libwayland, you can raw dog it over sockets manually. The idea is that there are standard protocols that can be queried and used,and you can implement this in any environment you want to. You could write the "frontend" in html and JS and run a wayland compositor on the web (which has been done [1]), you could do it with text or anything really, most people use some graphics stack.
- the compositor, of which there are multiple implementations (gnome, kde, all the wlroots compositors)
- the client, which often uses one of several toolkits (gtk, qt, several smaller frameworks, or even directly using the protocol)
- the wayland protocol (or rather protocols, because there are several extensions) itself
- other specifications and side channels for communication, in particular dbus.
Many issues (although I don't think the one in OP) are due to the protocol being underspecified, and so the client and compositor disagree about some semantics, or doesn't even have a standard way to accomplish something across all compositors.
If there were a single Wayland implementation in existence, I’d agree with your sentiment.
It's also the implementation most unsuspecting users will end up with.
With X11, it was simple: everybody used Xfree86 (or eventually the Xorg fork, but forks are not reimplementations) and libX11 (later libxcb was shimmed underneath with careful planning). The WM was bespoke, but it was small, nonintrusive, and out of the critical path, so bugs in it were neither numerous nor disastrous.
But today, with Wayland, there is no plan. And there is no limit to the bugs, which must get patched time and time again every time they are implemented.
For example Qt does per-monitor DPI just fine on X11; it's just that the way to specify/override the DPI values just sucks (an environment variable).
This stupid decision is going to chase us until the end of times since Xwayland will have no standardized way to tell its clients about per-display DPI.
This is something feasible on Wayland, X draws one large wide screen display.
In a way, Wayland in this case developed a solution for issue its creators brought into this world first
° For most programs.
Most people want to drag windown between screens and sometimes even split down the middle. One large display supports that much easier so that is what everyone switched to in the late 1990
X's "mechanism, not policy" has proven to be a failure. Without a consistent policy you end up with a pile of things that don't work together. "One big virtual screen" is a policy and it's one that works well enough.
Deciding what needs to be in the protocol and what should be in the application is never easy. It's important to be able to iterate quickly and avoid locking in bad practices. I think that's what Wayland tried to do by making everything a fine-grained extension, but it doesn't really work like that as the extensions become mandatory in practice.
Windows does this. Try to use in Windows 2 monitors with 2 different scalling factors. It is hit or miss. 100 and 150 works. 100 and 125 doesn't.
But you can definitely move windows to another monitor and Qt will use the right DPI for it. It is the same behavior as Wayland. "One large wide screen display" is exactly how Wayland works...
That, i think, is the main issue. Nobody wants to work with GTK1, or GTK2, or GTK3 anymore. Nobody wants to work with QT1, or QT2, or QT3 or QT4 anymore. Everybody wants the new shiny toy. Over and over again.
It is CADT all over. Earlier X was developed by an industry consortium. Now Wayland is a monopoly pushed by RedHat.
Pushed but still it seems it flails and takes too long to do the basic stuff
Can we trust them to "care" about desktops?
Oh wait. There is no more RedHat. There is IBM.
It was because of the crap LCD monitors (5 to 20 ms GtG) and how they are driven. The problem persists today. The (Wayland) solution was to render and display a complete frame at a time without taking into account the timings involved in hardware (you always have a good static image, but you have to wait).
I tried Tails (comes with some Wayland compositor) on a laptop. The GUI performance was terrible with only a Tor browser open and one tab.
If you do not care about hardware, you will, sooner or later, run into problems. Not everybody has your shiny 240 Hz monitor.
I can't tell if you are serious or not.
About 16 years old, for comparison, X is 40.
I've been hearing this for over a decade now. I don't get it. Just because xorg currently makes different clients aware of each other and broadcasts keypresses and mouse movements to all clients and allows screen capturing doesn't mean it has to. You could essentially give every application the impression that they are the only thing running.
It might seem difficult to implement, but compare it to the effort that has gone into wayland across the whole ecosystem. Maybe that was the point - motivating people to work on X was too difficult, and the wayland approach manages to diffuse the work out to more people.
I was really bullish on Wayland 10 years ago. Not so much any more. In retrospect it seems like a failure in technical leadership.
[1]: The people who actually developed Xorg are now working on various Wayland-related things.
I suspect it would be less challenging than writing a whole new wayland server.
Off the top of my head, I'd use a separate abstract domain socket for the window manager including some UUID, and then pass that to the window manager when launching it.
You could create these sockets on demand - one for each security context. On linux typically a different security contexts will either have different UIDs - in which case filesystem permissions would be sufficient - or they have different mount namespaces - in which case you make different sockets visible in different namespaces.
For SSH forwarding you could have SSH ask the X server for a new socket for forwarding purposes - so remote clients can't snoop on local clients.
> Good luck on your lonely[1] journey! > > [1]: The people who actually developed Xorg are now working on various Wayland-related things.
This is what I mean by a failure of technical leadership.
This is reminiscent of how Trusted Solaris[0] implements Mandatory Access Control (MAC) a la Orange Book[1].
[0] https://www.oracle.com/technetwork/server-storage/solaris10/... [1] https://public.milcyber.org/activities/magazine/articles/202...
SSH pretty much already does this. Per default (using -X) X11 forwarding is in untrusted mode, which makes certain unsafe X11 extensions unavailable. So remote clients already cannot snoop the whole keyboard input.
And then the window manager spawns whatever programs the user wants, and they end up sharing that $DISPLAY.
1. There is nothing fundamentally insecure about the design of X that couldn't be fixed with a bit of effort.
2. The effort required would be significantly less than what has gone into Wayland over the last 16 years.
Fixing the security of the system certainly would involve changes to more than just xorg. Anywhere there is a security boundary you'd need to make modifications - but that's OK, most of these security boundaries are younger than wayland anyway.
This discussion of $DISPLAY seems like distracting minutiae. Sure it's a medium size problem that would need solving, but that's all it is.
To address your specific point: it could be the responsibility of the window manager to set $DISPLAY to a less powerful socket when starting processes. Ultimately it doesn't matter though, because if the display server isn't doing some sort of sandboxing then the spawned process can just ptrace xorg and do whatever it wants. X being secure only matters in the presence of a security boundary and in that case it would be the responsibility of whatever is setting up that boundary to create a less privileged socket. Whether that be ssh or flatpak or whatever.
Once again, for emphasis: nobody has stepped up to do that, in the history of X (outside of proprietary products, of unknown quality).
Wayland is not some outsider project competing against X11. It's the people who were developing Xorg, saying they're fed up with X11 and need a clean slate to fix its problems.
Here's my unpriviledged user trying to ptrace my Wayland compositor running as me:
strace --syscall-limit=1 -p 697351
strace: attach: ptrace(PTRACE_SEIZE, 697351): Operation not permitted
In a world where the people doing the work switched to developing the Wayland ecosystem, arguing about a "could" that requires changing every window manager seems like a guarantee X11 will not get improved. Feel free to put effort into it...
> It might seem difficult to implement
This is exactly what OS did (my daily driver): https://forum.qubes-os.org/t/inter-vm-keyboard-isolation/315...
Can you explain who is forced to do what in that context?
Coincidentally I have got a graphics driver that likes crashing on OpenGL (AMD Ryzen 7 7840U w/ Radeon 780M Graphics)
Totally agree. The people saying "Wayland is a protocol" miss the point. Wayland is a protocol, but Wayland adoption means implementing stuff that uses that protocol, and then pushing it onto users.
Measure twice, cut once. Look before you leap. All that kind of thing. Get it working FIRST, then release it.
(Also, I don't use Wayland. I mean I tried it out but don't see any real benefit so I don't use it regularly.)
30 years ago X came with a server and some clients. Why it is so hard to do this, for wayland, today ?
Then write code.
Asahi Lina has demonstrated that a single person can write the appropriate shims to make things work.
Vulkan being effectively universally available on all the Linux graphics cards means that you have the hardest layer of abstraction to the GPU taken care of.
A single or small number of people could write a layer that sits above Wayland and X11 and does it right. However, no one has.
This argument don't make sense because Wayland started as a hobby and not to replace x11, was after it got traction, other people/companies started contributing that it matter
With complex projects like these that have to work well on such a wide array of hardware and configurations, lots of real world usage is required to achieve any level of refinement. Without widespread adoption, Wayland likely would have been stuck in experimental/toy status for much longer than it will as things are going currently.
Planes can be updated and repositioned without redrawing the rest of the screen (the regular screen image is on the primary plane), so moving the cursor is just a case of committing the new plane position.
The input latency introduced by GNOME's Mutter (the Wayland server used here) is likely simply a matter of their input sampling and commit timing strategy. Different servers have different strategies and priorities there, which can be good and bad.
Wayland, which is a protocol, is not involved in the process of positioning regular cursors, so this is entirely display server internals and optimization. What happens on the protocol level is allowing clients to set the cursor image, and telling clients where the cursor is.
Hopefully this general concern doesn't apply to Wayland and the "shape" you have described doesn't sound bad, but the devil is in the details.
The story is different for applications like games that hide the system cursor to display their own. In those cases, the client needs to receive mouse events from the compositor, then redraw the surface appropriately, all of which does go through Wayland.
Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors. Just the white cursor you browse the Web with.
Anyway, what you refer to is the legacy drm interface that was replaced by the atomic one. The legacy interface is very broken and does not expose new hardware features, but it did indeed handle cursors as its own magical entity.
The atomic API does support tearing updates, but cursor updates are currently rejected in that path as drivers are not ready for that, and at the same time, current consensus is that tearing is toggled on when a particular fullscreen game demands it, and games composite any cursors in their own render pass so they're unaffected. Drivers will probably support this eventually, but it's not meant to be a general solution.
The legacy API could let some hardware swap the cursor position mid-scanout, possibly tearing the cursor, but just because the call is made mid-scanout does not mean that the driver or hardware would do it.
> but it does add an average of 1 more frame latency
If you commit just in time (display servers aim to commit as late as possible), then the delay between the commit and a tearing update made just before the pixels were pushed is dependent on the cursor position - if the cursor is at the first line shown, it makes no difference, if on the last shown, it'll be almost a frame newer.
Averaging cursor positions mean half a frame of extra latency, but with a steady sampling rate instead of rolling shutter.
Proper commit timing is usually the proper solution, and more importantly helps every other aspect of content delivery as well.
I do believe it is a useful tradeoff, though.
Isn't it what many people refer to as "hardware cursor"? Is it possible for Wayland to rely on such a feature?
They just use the atomic API to move a cursor or overlay plane, which reflect how the hardware handles things. That the legacy API exposed a specialized cursor API was just a quirk of the design.
Note that planes are a power optimization more than anything else, as it allows e.g. the cursor to move or for decoded video frames to be displayed while GPU's render-related units are powered down. Drawing the cursor move, even though the render task is a rounding error, would require the render-related units to be on.
I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
It doesn't give a damn if you give it 2 buffers and one contains a mouse cursor and the other everything else or if you give it 2 buffers and one is everything including the mouse and the other is a video, allowing complete power collapse of the GPU rendering units.
Often they support more than 2 of these as well, and with color conversions, 1D & 3D LUTs, and a handful of other useful properties. Mobile SoCs in particular, like your typical mid/high end snapdragon, actually have upwards of a dozen overlay planes. This is how Android manages to almost never hit GPU composition at all.
On desktop linux all of these go through the drm/kms APIs.
Mobile chips are as you mention far ahead in this space, with some having outright arbitrary plane counts.
Even though it's only 3 planes, they are relatively feature-rich still. In a typical desktop UI that would indeed be primary, cursor, and video planes. But if the system cursor is hidden, such as in a game, that frees up a plane that can be used for something else - such as the aforementioned game.
On AMD in particular, the cursor plane must match several aspects of any plane it overlaps with, including transform and color pipeline IIRC.
The AMD SoC in my laptop (much newer than the steam deck) only exposes two overlay planes to share among all 4 display controllers. Intel used to have a single overlay plane per display.
The Raspberry Pi 5 on the other hand intentionally limited the exposed overlay planes to "just" 48, as it can be as many as you have memory for.
You can peek at the reported capabilities of various devices here: https://drmdb.emersion.fr/
What I meant here is that I didn't see why asynchronous updates may introduce tearing; but my tired brain couldn't quite formulate it properly. And to answer that, it's clear to me know that an update of the pointer position while the pointer sprite is being drawn would introduce a shift somewhere within that sprite, which is, I suppose, the tearing discussed (and not a whole frame tearing).
> I don't know however if the mouse pointer picture is still handled the VESA way, or if GPUs video cards nowadays have a more generic API, or what.
Also, the VESA interface doesn't seem to handle mouse pointers, it's something that was available in the VGA BIOS, to provide a uniform support for this feature, as each vendor most likely did it their own way.
To the user that's an irrelevant distinction.
I also don't think this matters that much - with X11 this was optimized in one place by people that care about such details while with Wayland now every compositor developer (who in general are much more interested in window management policty) needs to become a low leve performance expert.
> Second, just to be clear, this only discusses mouse cursors on the desktop - not the content of windows, and in particular not games even if they have cursors.
Games can and sometimes do use "hardware" cursors as well - after all, they also care about latency.
When updating the cursor position, check if line being output overlaps with the cursor. If it isn't, it's safe to update the hardware cursor immediately, without tearing. Otherwise, defer updating the cursor until later (vblank would work) to avoid tearing.
Of course, this assumes it's possible to read what row of the frame buffer is being displayed. I think most hardware would support it, but I could see driver support being poorly tested, or possibly even missing entirely from Linux's video APIs.
But given that different displays work differently, I'm not sure it would worth the hassle.
> Wayland, which is a protocol
This is wayland's biggest weakness. The effect is diffusion of responsibility.
I'm not. My comment doesn't address the latency of gnome shell. I understand the boring technical distinctions between wayland and wayland client libraries and wayland display servers and gnome shell and mutter and sway, blah blah blah. Much like I understand that Linux is a kernel. That it is inspired by UNIX, but it is technically not a UNIX. I also understand that if someone describes themselves as a Linux user they probably don't just mean that they have a Android phone or that the display controller in their dishwasher or wireless access point happens to include Linux the kernel.
The "well acksually wayland is just the name of the protocol" that emerges whenever a problem is brought up is a symptom of the underlying problem with wayland the system. The confusion that gives rise to these deflections is also a symptom of that problem.
By Conway's law systems end up resembling the organisations that produce them. In this way the design of wayland the system seems to be designed by people who don't want to work together. I can see a parallel with microservice architecture.
Imagine comparing HTTP1.1 vs HTTP3. These are protocols, but in practice one compares implementations. I can pick curl for http1.1, but Python for http3, and http3 would very likely measures as slower.
Is that the protocols fault?
I very much doubt that Wayland makes a difference for this test; Wayland is for IPC between the client and server. Moving the cursor around is done by the server, without needing to talk to the client.
> well acksually
It’s not the protocol’s fault, but the system and organisation that brought it.
I know this because I worked at a place which was entirely based around microservices using Rest HTTP as the protocol/interface, some teams had to use alternate methods because HTTP protocol was the bottleneck.
I do not. Does anyone know a good but quick introduction into these concepts and the problems they cause or fix?
All I know is that Wayland was supposed to be faster by taking some slow parts out of the loop, but that doesn't seem to be working, according to these figures.
I would just rather pay someone 100 bucks and not care whose fault it is. (And yes, that's why I don't use Linux on the desktop anymore, as much as I loved i3 and tiling environments.)
It's probably in a hundred places that all add up, all of which are no particular person's responsibility. So, that probably means that 10 people from 10 different projects need to get on a call or mailing list together and find a plan of attack.
But it won't happen. And people will keep wondering why Linux can't ever get a foothold on the desktop.
Maintained: Only for bugfixes, except by metux, who is working on starting a fork.
This isn't a big issue because Xorg splits a basic GUI into dozens of different daemons and services (compositors, window managers, input daemons) so it seems like there are different X11 implementations even when there's usually only Xorg.
Every time someone complains about wayland there's someone informing you how it achtually it isn't.
The organisation of Wayland sounds great, but it is very hard to share optimised code between compositors since key parts that affect performance (in this case latency) are largely developed outside of any shared library code.
The "organisation" of Wayland reminds me of the UNIX wars; this is going to get worse before it gets better.
SVR4 Wayland anyone?
xref the time it has taken the Rust rewrite of the GNU coreutils and arguably coreutils is a much easier problem.
The "problem" with this in Wayland is that before people 'ran Xorg with GNOME on top", now they just run GNOME the same way they run Chrome or Firefox to use HTTP - it will take time for people to get used to this.
I think the answer is also Wayland. It's very confusing because there is wayland the protocol and wayland the system, where the protocol is a part of the system - but there is no separate name for it. When people discuss wayland they are very rarely talking about wayland the protocol, and are much more likely talking about wayland the system.
In the metaphor of a web server and a web browser, Wayland would be the HTTP specification. What you're usually interested in is what server you're running, e.g. GNOME's Mutter, KDE's Kwin, sway, niri, or what client you're running, e.g. Gtk4, Qt6, etc.
Part of the problem will doubtless be the USB (and Bluetooth) stacks including the device hardware and firmware. When keyboards and mice were serial devices with their own interrupt making the code path fast was achievable. I'm not so confident that modern peripheral stacks can be made to run with the same priority. It becomes even more challenging for devices sitting on a bus with multiple device classes or multiple protocols between the device and driver (USB -> Bluetooth -> Mouse).
I hope devices can be sped up but we're a long way from a keypress triggering an interrupt and being handled in tens of milliseconds[0].
In that compartmentalization there was nothing else competing for attention. You didn't get USB bus contention because there was just the one mouse, ever, on the line between the user and the X11 server in the X terminal.
Alternatively you move straight up to USB 4, where we get pci-e interrupts again! :)
In theory, yes. Not so, in practice.
> USB 3.1 signals at 10 GHz.
Yes but a mouse works at 12 MHz.
Tearing would affect everyone that uses a computer with X11 but your proposed example of a TV with 30i refresh rate would only affect the tiny subset of users that use a CRT television as a monitor, right?
I'd say: it really doesn't matter at all, in fact it's even more argument to make the pointer as low latency as possible. The mouse pointer is your control element as the controller of the interface. The UI element (and everything else onscreen) is the computer's element. It is an interaction between two entities and there is no need to introduce an expectation that the computer be "zero frame delay" on UI elements. It certainly sucks if it's more than a few frames for it to react, but so be it. However for the mouse cursor itself, it should absolutely be immediate because it is the user's representation on this virtual surface, and as such should follow inputs without being unnecessarily impeded. (Case in point: some UIs intentionally add delay or effects [like a fade-in] to UI element response. It's a… "choice"… but sometimes it works.)
Ultimately, the mouse pointer is an extension of hand-eye coordination; the only thing worse than making it lag (which you can learn to adapt to, albeit it will be very annoying) is to make it lag inconsistently (which you can't adapt to and will grow frustrated with extremely quickly.)
And, unfortunately, from personal experience, the mouse pointer on Wayland gets choppy if system load is high. X11 was significantly better with this; with Wayland the compositor [kwin_wayland to be specific, might be a KDE issue] doesn't even seem to be "unnice" or have some kind of scheduling priority over other tasks — I'm not sure if I'm missing something there but that seems to be a glaring oversight on its own.
IIRC it all started going downhill in Xorg when glamour appeared. After the cursor rendering path wasn't async-safe for execution from the signal handler (which something opengl-backed certainly wouldn't be), the latency was worse.
I remember when even if your Linux box was thrashing your mouse pointer would stay responsive, and that was a reliable indicator of if the kernel was hung or not. If the pointer prematurely became unresponsive, it was because you were on an IDE/PATA host and needed to enable unmask irq w/hdparm. An unresponsive pointer in XFree86 was that useful of a signal that something was wrong or misconfigured... ah, the good old days.
That's still the case. The mouse cursor is rendered with a sprite/overlay independent from the window composer swapchain, otherwise the lag would be very noticeable (at 60Hz at least). The trickery starts when dragging windows or icons around, because that makes the swapchain lag visible. Some window composers don't care any longer because a high refresh rate makes the lag not as visible (eg macOS) others (eg Windows I think) switch to a software mouse cursor while dragging.
Cool. Never really thought about why they don't exactly keep up with the cursor.
I like what's being done here but we need a more thorough experiment before we can start drawing conclusions.
It's not really hard to guess at - it's probably caused by abstraction layers resulting in multiple layers of buffering. Of course, it's all just speculation until proven by experiment, so good on TFA for doing some of that.
Retro hardware from the C64 or NES era was single-tasking with such predictable timing you could change the video resolution partway through the screen refresh and have two resolutions on screen at once. If you want, you can check the player input and update Mario's position right before he gets scanned out - the minimum possible latency by design is zero frames. (Of course, in practice the whole screen data is buffered up during vblank, which is the same latency as using a framebuffer. Also the NES didn't allow changing video registers outside of blanking periods. But the C64 did.)
X11 isn't that close to the metal, but it's designed with a single level of buffering (assuming a compositing window manager isn't used). There is a single framebuffer covering the whole screen. Framebuffer updates are completely asynchronous to scanout, so there is around 0.5 frames of latency on average between when a pixel is changed and when it's scanned out. Apps borrow pixels from this big global framebuffer to draw into. An app sends a "draw rectangle" command relative to its own borrowed space, and the server calculates where the rectangle should go on the screen, and draws it into the framebuffer, and then with an average latency of half a frame, that part of the framebuffer is scanned out to the screen.
On Wayland, there are more layers. The app has to draw into its own pixel buffer (noting that it is probably double- or triple-buffered and has to redraw the entire window rather than relying on non-changing stuff to still be there) and then once the compositor receives the buffer from the app, it has to do the same thing again, copying all the per-app buffers into a big screen-size buffer, which is probably also double- or triple-buffered, and once the vblank hits, the most recent full frame is drawn to the screen. It's just more work with more layers, and there's no surprise that latency is higher. It's hardly the first or 100th time that an elegant software architecture resulted in the computer having to do more work for the sake of keeping the architecture pure.
Something important to note is that tearing reduces latency. If new data is available while the scanout is partway through a frame, you have the choice to draw it immediately, causing tearing, or wait until the next frame, increasing latency. Always doing the high-latency thing is an explicit design goal of Wayland. (A third theoretical choice is to always time things precisely so that the new frame is calculated right before the end of vblank, but... in a modular multitasking architecture with apps written by different organizations and unpredictable CPU load, good luck with that.)
Now the really big caveat: I'd be surprised if GNOME's X11 WM wasn't compositing. If it's compositing, then X11 behaves similarly to Wayland but with even more IPC and what I just said is no reason it should have one frame less latency on average. Still something to think about though.
> With my 144Hz screen,....Wayland, on average, has roughly 6.5ms more cursor latency than X11 on my system...Interestingly, the difference is very close to 1 full screen refresh. I don't know whether or not that's a coincidence.
The fact that the latency is almost 1/144th of a second means that it might become 1/60th of a second on standard 60Hz monitors. This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
My guess: the "true" numbers are close to 2.5 (half a frame of random phase of when the mouse is touched vs. refresh, plus 2 frames to move cursor) and 3.5. If you throw out the low outlier from each set you get pretty close to that.
(of course, the 125Hz mouse poll rate is another confound for many users, but this guy used a 1KHz mouse).
> This is hard to notice consciously without training, but most people can "feel" the difference even if they can't explain it.
Yah. 7ms difference is not bad vs 16.6ms is starting to be a lot.
IMO, we should be putting in effort on computers to reach 1.6 frames of latency -- half a frame of random phase, plus one frame, plus a little bit of processing time.
[1] https://raphlinus.github.io/ui/graphics/2020/09/13/composito...
[2] http://number-none.com/blow/john_carmack_on_inlined_code.htm...
Put another way, a system fast enough to composite at 144Hz should be able to composite at 60Hz while only allocating 1/144 seconds to the compositor, which would require offsetting the presentation times as seen by the compositor’s clients by some fraction of a frame time, which doesn’t actually seem that bad.
It gets even better if variable refresh rate / frame timing works well, because then frames don’t drop even if some fraction of compositing operations are a bit too slow.
I assume I’m missing some reason why this isn’t done.
We need to cut some deadline and doing it at vsync is the easiest way
A strong argument that it's never more than one frame's worth of latency.
I think there may still be the issue that many compositor/GPU combinations don't get hardware cursor planes, which would definitely cause a latency discrepancy like this.
Lol no. X is from 1984: https://www.talisman.org/x-debut.shtml
That means Wayland is only 66% the age of X when Wayland came out, or you'd need 50% more of Wayland's life before its as old as X was.
Approximately 99.999% of those complaints are uttered by people who 1) are not doing the work, 2) are not interested or capable of doing the work, and 3) do not understand or refuse to acknowledge that the Wayland and X developers are _mostly the same folks_ and they _do not want to work on X11 anymore_.
I don't really have a stake in this argument, except I'm bone-tired of seeing people whine about Wayland when they're using software somebody else gave them for free. There are enough Wayland whiners that, by now, they could've banded together and started maintaining / improving X and (if their complaints and theories were correct) left Wayland behind.
Strangely -- even though X is open source and eminently forkable (we know this, because XFree86 -> X.org) it gathers dust and none of its proponents are doing anything towards its upkeep.
When someone shows up with "Wayland isn't as good as X, so here is my modernized fork of X anyone can use" -- I'll be quite interested.
This is a fair point, but the quality of Wayland isn't really at issue for this -- Wayland could be much better than X by all accounts and it would still require you to rewrite software for it. (Assuming XWayland doesn't suit your needs, anyway.)
You're far more likely to get people to expend the requisite time and effort if it's indisputable that the end result will be worth it.
> it would still require you to rewrite software for it
Does Wayland support the entire Xcompose system or an equally powerful alternative?
Wayland does genuinely sound interesting, if flaky and half-baked (it reminds me of systemd in that way). Certainly, the world doesn’t owe me a port of my window manager. But if the Wayland transition forces me to use GNOME then my life will be worse.
> if flaky and half-baked (it reminds me of systemd in that way)
They already have. X is already more full-featured and stable than Wayland (yes it is missing certain niche features that Wayland has). Sometimes the most important thing you can do with a piece of software is not screw around with it.
Eh maybe. Ultimately you can never be sure something is possible without doing it.
I'm pragmatic here, if and when Wayland offers me a better experience than X then I'll use it. I just resent distros etc. pushing me towards it when it's currently a downgrade.
...Yes? Wayland being a regression in terms of features and bugginess is kinda a sticking point.
> Approximately 99.999% of those complaints are uttered by people who 1) are not doing the work, 2) are not interested or capable of doing the work, and 3) do not understand or refuse to acknowledge that the Wayland and X developers are _mostly the same folks_ and they _do not want to work on X11 anymore_.
If X works for someone and Wayland doesn't, none of that matters. It doesn't matter how much you insult X users, it won't make their usecases invalid or make Wayland good enough.
> I don't really have a stake in this argument, except I'm bone-tired of seeing people whine about Wayland when they're using software somebody else gave them for free.
It cuts both ways: Users aren't entitled to free work, and developers aren't entitled to their work being well-regarded. Giving software away for free has never meant that people can't point out its problems.
I agree that X11 has some faults, and I really want Wayland to work, but everytime I try it, something ends up being broken.
It's fine for you to not care about that usecase, but other people do care and that's also valid. There even exist devices that mostly exist to watch video and we'd really like Linux and some standard graphics stack to work on them.
My datapoint was just in reply to someone writing that tearing under X has not occurred for 5 years.
Honestly Wim did this right on the audio side. If it was so important to bring what Wayland has brought someone should have written a display server that talked to display clients just like the old server did and upgraded gracefully if the client had the new capabilities. You didn't even have to recompile software to use pipewire, let alone rewrite it.
True, but sandboxing can fix your browser being able to read your spreadsheets without asking, or your spreadsheet having network access. Just because you have to work with programs and data doesn't mean they're all in the same security domain.
Now, you might not notice the tearing if you've got V-sync properly configured. But V-Sync doesn't stop screen tearing, it just drops torn frames and decreases your refresh rate until your system can keep up with demand. It's an ugly hack on laptop hardware and an even uglier one on the desktop.
I don’t believe that I’ve ever noticed or been bothered by static tearing, but maybe I’m just used to it because I’m old and displays have always had it?
While true, this statement is also useless. On a meta level, what you're essentially saying is that users' rights to feel resentful, is more valuable than achieving an outcome.
The only relevant problem with X11 to me is that all of its developers are stopping to work on it and it won't run on any new hardware soon (and probably on old hardware too.)
The only way to achieve the outcome you want, is for someone to do the work. They're not going to do it. If you'd rather keep being resentful rather than do the work towards your own claimed desired outcome, then apparently you don't really value the outcome you so claim to desire; you value being resentful more.
It's more useful than calling X users "whiners" while ignoring their complaints. And no, I'm pointing out that the users are resentful because they're being pushed to an outcome that's worse and therefore not desirable to achieve.
Not talking about the technical points of the whole X/Wayland considerations here, but a group of people in such debate is always as vocal as they'll do nothing and put unreasonably high expectations on other people to listen to them and have no choice but to agree. The fact that X.org's core team initiated Wayland and stopped X.org development is disregarded: these people Will Be Right against literally everything, they can't be reasoned.
My previous coworker was fired for gross incompetence in the end yet he never acknowledged anything wrong. He always knew better than literally the whole world that bash scripts run manually where superior to Terraform for the whole company's infrastructure. Yes, he was that stupid. I guess this is why he ended lying like hell on his LinkedIn profile, concealing he had had four employers in four years.
Seeing technical debates with such poor quality is worrying. If people with quantitative minds fail to entertain a fact-based discussion, how can we expect our political life to acknowledge there is only one reality and stop lying?
Before you voice your outrage, here are some facts that won't dis-exist just because you disagree with them:
- https://www.phoronix.com/review/x_wayland_situation
- https://ajaxnwnk.blogspot.com/2020/10/on-abandoning-x-server...
Are you saying that Wayland is only for developers? Are people not allowed to complain when The New Thing turns out to be less good than the supposedly Obsolete Thing?
> 3) do not understand or refuse to acknowledge that the Wayland and X developers are _mostly the same folks_ and they _do not want to work on X11 anymore_.
I'm fully aware of that. I understand X11 has its limitations, and some of the goals of Wayland sound very appropriate for the modern PC environment, but if after 16 years, there are still many situations where Wayland does worse than X, that's not a great sign, and it will make people continue to use X.
My primary complaint with Wayland is that it is a textbook example of Brooks’s second-system effect. It forsook backwards compatibility (yes, there’s an X11 server but to my knowledge there is no way to just run Wayland and one’s X11 desktop environment).
> Strangely -- even though X is open source and eminently forkable (we know this, because XFree86 -> X.org) it gathers dust and none of its proponents are doing anything towards its upkeep.
I suspect that is because the X/Wayland guys have sucked all the oxygen out of that particular room. A newbie shows up and is told that X.org is legacy and he shouldn’t work on it, so … he doesn’t.
And of course X.org really is a bit of a disaster due to being written in C.
Wayland made some huge design changes, and those changes all have pros and cons. How do we tend to communicate about pros and cons? By complaining.
It sucks that Xorg's accessibility features are so baked-in that you must be compatible with them; because that demand can become a wall of assumptions. It sucks that Wayland leaves 100% of accessibility to the compositor; because that creates a lot of redundant work, and the best implementations get siloed into plasma or gnome, creating multiple smaller instances of the tight integration that made UX less flexible to begin with.
Using Xorg, you can run an incredibly lightweight window manager that never even thought about managing keymaps, adjusting mouse acceleration, setting the display resolution, etc., because that's not it's job. I can get absolutely everything I want from i3wm on Xorg... except fractional scaling. So far, it looks like plasma is the only one that has really figured that one out, and I still get a tiny pointer sometimes.
The best thing about free software is that we can choose how it works. The more tightly integrated the pieces are, the fewer choices are available. Fewer choices results in more complaining. Wayland may be the most significant instance of this pattern in Linux history. The second most significant instance was already present: Desktop Environments.
The biggest source of complaints I have is that the best aspects of plasma are trapped inside KDE. If you want the best Wayland compositor out there, you have to have it wrapped in an entire full-featured Desktop Environment.
---
Complaining is the best thing about Linux, so long as it's able to drive real change. The more stubborn and inflexible a system (and whoever designs it) is, the more disconnected complaints become. That's the real problem that deserves our attention.
It's clarifying if you simply say "users."
That's literally it.
Because there was a big fracture and doubt in the community support for Wayland, not to mention alternatives like mir popping up and diverting resources.
Do you see how some might think it wasn't the best strategy?
In fact, in almost 2 decades of using Linux every day, I can't remember X doing anything similar. It's always just worked.
Well, we have very different memories then. Sure, X worked reliably once configured. But configuring it was a marathon in hell, as per by my memory, and all the litany of forum posts crying out for help all across the internet. Like, I have at one point had to rescue an install by changing back the config file without a screen at all!
I also fundamentally disagree with the idea that X "does too much", which is often cited in favour of Wayland. The fact that X encompasses everything about how I interact with my computer is one of the things I love about it. I might switch WMs and DEs, but my xmodmap, xresources etc remain the same.
It hasn't needed any manual configuration for a normal setup in ages.
There’s nothing Wayland-specific that would introduce this latency, so I wonder if wlroots/sway have any additions lag.
After upgrading to plasma6 from 5, all the desktop animations have started stuttering. Probably your hardware is too new.
Hell, it's been 12 years and not a single wayland compositor supports screen readers for the visually disabled yet. They are toys.
One of the really cool things that has come out of this is seeing how people are adding accessibility workflows on top of the ecosystem. There is one person who controls their entire desktop via the tiling window manager using voice control![2]
So no good for games, no good for professional graphics, no good if you don't see well... basically no good if you're any different from the people who hacked it together.
But, hey, with any luck they cut down on the screen tearing that I've never noticed.
Why would you expect it any different? How can one implement things that they have no need or no hardware for? The entitlement is a bit jarring.
Also I think they merged something last year: https://github.com/swaywm/sway/pull/7681
[1] https://gitlab.freedesktop.org/wayland/wayland-protocols/-/m...
That being said, thanks to Nvidia I never got colour calibration to work right in X11 either, so I have no horse in this race. Would be cool to finally get HDR working for the first time, but I don't know if that'll ever happen on Linux with Nvidia hardware. Guess I should keep dual booting for HDR content until either Windows 10 dies off or I get new hardware.
I do actually notice the lack of tearing in Wayland, especially under heavy load. Used to annoy me to no end to see my windows tear when just dragging them across a 1080p screen. I don't know if it was an Intel driver bug (I tried all the config settings) or something X11 specific, but Wayland did finally fix the tearing issues I had.
I haven't noticed any problems with colours in either Gnome or Gamescope (except for the lack of HDR, of course, but that's also true on X11) so whatever is causing issues for you seems to be DE specific. Looks like we both have issues other people never encountered, that's what makes graphics stacks so impossible to debug and code for.
Color calibration can absolutely be retrofitted into this (versioned) protocol, and there is work ongoing.
It feels like you're probably blaming the wrong people here. You should look at the companies that make this peripherals that don't also offer you linux drivers.
The only hardware I know for sure is lacking is stuff like Stream Decks, but that kind of hardware is difficult to generalise for.
Plus, most working hardware that doesn't fit a standard HID protocol already has drivers in the Linux kernel written by manufacturers.
I know that Georges Stavracas has been working on a controller for the Elgato devices for a while (https://flathub.org/apps/com.feaneron.Boatswain) though it needs direct access to the devices instead of through some generic driver (https://gitlab.gnome.org/World/boatswain#udev-rules).
...Which I don't get because the Xe driver is said to explicitly support, at minimum, Tiger Lake. I played Minecraft on the iGPU with Xe and it was perfectly fine. It... drew 3D graphics at expected framerates.
IDE text cursor delay? Unacceptable. Shell text cursor delay? Unacceptable. GUI mouse cursor delay? Unacceptable.
All are immediate deal-breakers for me.
I recently had a problem with extreme mouse slowdowns on KDE/Plasma 6 on X11 with NVidia. I noticed the slowdown was particularly extreme over the tabs of my browser.
The fix, in case anyone else also has this problem, is to disable OpenGL flipping. I have no idea what OpenGL flipping does, but disabling it (in nvidia-settings and in /etc/X11/xorg.conf) fixed the problem.
I'd love it if someone could explain why. Or why this isn't the default.
Even System 1 on almost infinitely less powerful gear 40+ years ago super-prioritized cursor responsiveness, outshining all competitors. The OS and UI underpinnings are now 100% different, but the priority remains.
Those curves work fine for the Magic Trackpad, but they should not come default for any desktop Macs that use mice as their primary input.
When compositors (even with X11 composite) and GPU renderers becamse fashionable, the latency became awful, literally 100s of msecs. Which was weird considering, I grew up playing DOS games that didn't have this issue. Things improved over time but it's not clear to me if we ever got back to the golden days of CPU rendering.
The two sample t-test statistic is -4.74 with a p-value of 4.20e-05.
I've long wondered why some terminals and operating systems felt like they were pretty laggy. In the sense that if I press my character to move, how long until they actually move?
iTerm2 on MacOS is worst offender. But I can't tell if it's iTerm2 or MacOS itself. I remember trying other terminals on Mac, and it was mostly the same. iTerm2 itself also had bunch of rendering options, but I couldn't get them to do anything I could actually feel affected input lag. One thought I had was: maybe iTerm2 added 1 frame lateness, and MacOS itself added another? But that would be ~30ms on a 60fps monitor which I can easily already tell on a server-connection NetHack.
I also have no idea if measuring "N frames late" is actually a sensible way to think about this. I assume computers know when a key is pressed at a higher frequency?
Linux xterm and rxvt were the fastest that I remember, on X11 (I've never used Wayland like in the article, well not for NetHack anyway).
I have no idea how compositors work on any of these operating systems or X11/Wayland to say anything smart about them or why they would be slower or faster.
Reading the article...I realized I have that same iPhone 15 Pro with 240fps recording. I could replicate that experiment, but testing out terminals on various operating systems instead I used to play on. So I could now test if my gut feeling was right all along or not. or maybe it lied to me all these years.
I wrote the idea in the article down on my notes, maybe when I'm bored enough I'll try it :) and I can stop saying propaganda about iTerm2 being slow if it turns out I was completely wrong. Maybe I have easier time because all my NetHack days were 60fps, so I have more leeway in measurements.
I'm not active in NetHack playing anymore, although I sometimes play it as a comfort game.
There, it's distinct in that other terminals have lower latency on the same system.
"When people measure actual end-to-end latency for games on normal computer setups, they usually find latencies in the 100ms range."
That is really surprising to me. 100ms is HUGE. On NetHack, sshing into servers the difference between 50ms and 150ms is enormous. If do my experiment and I'm not too lazy I want to check on that 100ms figure, your link points to more sources which I didn't check on right now.
I don't know if I'm blind but I can't tell when that article was written exactly. But it mentioned MacBook 2014 and Lubuntu 16.04 so maybe it's around mid 2010, 2016 possibly? (I should check properly if I do my experiments)
The author in the Wayland vs X11 used mouse flicks on the video. Just now while writing I kind of had a thought if this needs some more deep thinking: how do I accurately measure "my finger touched this button and X milliseconds later character moved". Wondering what to consider as the "finger touched this button" event in a 240fps video. Maybe I can do what the author did and map a key to a mouse instead, because maybe there's a much-easier-to-see physical movement with a mouse. But then, am I really measuring faithfully my old NetHack-like environment? Or maybe this isn't really an issue and it'll be easy to tell.
Too much for my small brain to think in the moment :) that link gave me at least some validation that iTerm2 maybe really is a bit slow, or at least was at that time. There's also bunch of terminals on that page I've never heard of.
Today iTerm2 is my daily driver, but xterm is still really fast on my Fedora Linux system I have on the side.
Accessibly is far worse to the point of embarrassment. Latency is worse. Compatibility is gone. Everyone in the ecosystem needs to do more work to end up with fewer features for end users.
This is the opposite of systemd in every way. Systemd upset a lot of people because it was entirely user focused and didn't care about the beauty of the architecture. Wayland is only obsessed with how nice their apis are, not what the end user experience is.
Let's end this disaster. It's been 16 years. It's almost old enough to vote. When will we accept it has failed?
So what is your proposal, starting again from a blank sheet?
I don't think it will help.
Still works great for me!
Fine, we are all happy then.
Use what you think is better then.
If you feel this way I recommend watching this talk by Daniel Stone from linux.conf.au 2013, The Real Story Behind Wayland and X https://www.youtube.com/watch?v=RIctzAQOe44
X sucks for how graphics work in the 21st century, there's a big reason Google didn't use it for Android and instead made something that's actually more akin to the Wayland model .
(I do not suck, my setup had 100ms lag)
You can learn to play Space Channel 5 by tapping 100ms ahead of the beat. But I worried that if I succeeded too much, I'd end up doing everything that way: jumping red lights, taking espresso shots while they were still being poured etc.
...what it's like to experience video game lag in real life.
https://bugzilla.gnome.org/show_bug.cgi?id=745032 https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/749
I also find Wayland more laggy and more buggy without concrete evidence of course.
It has significantly improved lately though, far more stable than it used to be.
Not arguing though, you are right it is just issues: Drivers, the Wayland implementations, how some plethora of apps and libraries have been battle tested then optimized, years over years, for X11. Not as much for Wayland display yet.
I think it's safe to assume that it is actually simpler, given that we already have multiple Wayland implementations, but still basically just the one X11 server implementation. Can one or more of those implementations shave off one or two milliseconds overs the next 40 years... Probably yes.
footnote: there is something wrong with the commonly used web search engines and I am unable to find that X11 web demo, I think they are prioritizing recent content over good content. you would think with how interesting that web demo was it was it would show up right away. but I got nothing, so no links. Anyway it was a deep dive into the intricacies of X11 window painting and the author had ported a limited subset of an X server to javascript in order to provide a live demonstration of the topic. I will keep looking.
Found it. https://magcius.github.io/xplain/article/x-basics.html
It is a common problem with these "simple" things. The problem is complex, and if you simplify one part, you are just pushing complexity elsewhere, you are not making it disappear. It is sometimes a good thing, but in the case of Wayland, it didn't go well.
In any case wayland is not bad if you only have pretty basic needs I guess, some basic things look easier to me there from a user perspective, and troubleshooting x11-related issues for a non-technical person is no fun either.
It was designed to fix tearing issues, not latency issues.
But then the Wayland designers found out that players often prefer tearing to minimize latency, so the tearing protocol was implemented.
When a committee of perfectionists, rather than real people or companies, design something, you often get something that is completely unusable by anyone but that committee.
And that's exactly how it's played out so far. Wayland is still largely incomplete, not to mention it doesn't even have a reference implementation [1], and still doesn't include/describe several essential desktop APIs and features, which results in this protocol not having a ton of universal tools and applications that work across all of its implementations, including but not limited to:
* Keyboard layout switching and input management
* Display configuration
* Clipboard management
* Tools to manage windows/automate tasks
* Lots more
1. https://gitlab.freedesktop.org/wayland/wayland/-/issues/233
It might be a better technical design to have the other stuff outside of the display protocol. Just because Xorg implemented something does not mean you have to put it in the Wayland protocol.
In any of these cases there may be one or more daemons behind the scenes handling the “raw” input—possibly even in cooperation with kernel-level pre-processing code, to ensure low latencey—but most event delivery to applications is associated with windows, with options to get lower-level access if needed.
One of the things that helps many of the systems described above with latency is kernel participation, whether by pushing much of the preprocessing of events down to the drivers so there’s little for userspace to do, or by implementing kernel-level zero-copy IPC (e.g. use of Mach messages by NeXT and Apple).
If human interface IPC happens entirely in userspace and requires multiple context switches to get an event from device to a display change, you’ll wind up with hitches and delays unless there’s some sort of scheduler hinting that ensures each stage in the pipeline runs immediately after the last.
This is, of course, why there was a lot of desire by Wayland advocates for kernel dbus support, but they went at the problem backwards: “Let’s take DBus, and make it fast by putting it in-kernel,” *without* first trying to make it as fast as possible without kernel support, *and* without trying to figure out the minimal feature set for kernel-level IPC that would be needed to support it (which may not look like DBus).
How is that bad design?
Actually is was designed because the X11 codebase was bad and nobody wanted to work on it.
This sounds like "Wayland codebase is good and everybody wants to work on it".
Wlroots exists.
Good luck using it as your graphics subsystem.
> Typometer works by generating OS input events and using screen capture to measure the delay between a keystroke and a corresponding screen update. Hence, the measurement encompasses all the constituents of processing latency (i. e. OS queue, VM, editor, GPU pipeline, buffering, window manager and possible V-Sync).
I created small application with raylib which would toggle the screen from black to white when clicked. When the screen changes from white to black, the voltage across of the photo diode would drop, which would create a falling edge I could trigger on using the oscilloscope.
Test Setup:
I ran the following tests on my AMD Framework 13, under CachyOS with all packages up to date. For each test I set the Raylib application to run at 800x600@60Hz and left the photo diode in place near the bottom right corner of the screen. For each test, the display output scaling was set to 1x (2256x1504), and variable refresh rate was disabled. Using the oscilloscope I measured from the first trigger buffer sample that the voltage of the button reached steady state to the first sample where the voltage on the photo diode dropped below steady state to 1.5v
Caveats:
The times seem high, but they encompass the entire chain of Push button -> App rendering next frame.
There is some loss in time resolution because of the abysmally small size of my cheap oscilloscope’s trace buffer.
I only collected 9 samples for each configuration because I’m clumsy on the controls of the oscilloscope and this took a little bit.
The raylib application was running under Xwayland for the Wayland tests which somewhat skews the results.
KDE Plasma (Wayland): 181ms, 188ms, 139ms, 148m, 137ms, 156ms, 163ms, 164ms, 151ms => Avg 158ms
Sway (Wayland): 172ms, 175ms, 168ms, 153ms, 144ms, 152ms, 161ms, 172ms, 156ms => Avg 161ms
I wouldn't use that. Try KDE if you want a decent gaming experience. Gnome was very slow in even implementing something like adaptive sync, so they hardly are prioritizing latency issues for the end user.
I.e. Gnome's problems ≠ Wayland problems in the broader sense. Conflating the two is wrong.
This means Gnome's problems = Wayland problems.
Furthermore, the latency issues with Wayland are inherent. Compositors will always have worse latency than writing directly to the front buffer.
Gnome is big, but it's not the most popular DE out there, so it can't be treated even as a de-facto standard in the Wayland world. Some things might be treated that way, but they are lower level (let's say Pipewire, libinput etc.).
And regardless, it can't be used an example that blames Wayland for Gnome's issues, unless literally every single compositor has this problem and the real reason is Wayland's inherent limitations. There are such limitations in general indeed due to Wayland still evolving, but this isn't one of them.
This is not really true, if a full screen application is detected, the compositor can tell the GPU to instead just display the contents of the applications buffer, skipping the compositor.
IIRC this is broken on Xwayland GNOME.
I mean... in the sense that "ALSA has to be faster than Jack ultimately", I guess; that fact may not be useful for a given setup
Anything related to the compositor won’t affect the game, unless it’s in windowed mode, then there can be some strange interactions.
In windowed mode they can be captured, then things get tricky, but in full screen they draw direct to the gpu, vulkan itself is completely headless.
Gamescope's direct scanout bypasses this.
I wasn’t aware that this changed, but you could be right. Its definitely the same on Windows as it always was, which is the platform I most developed games for.
Not really. Most games use "borderless windowed" mode instead of fullscreen exclusive, and even FSE is not true exclusive mode anymore in most cases.
https://devblogs.microsoft.com/directx/demystifying-full-scr...
> When using Fullscreen Optimizations, your game believes that it is running in Fullscreen Exclusive, but behind the scenes, Windows has the game running in borderless windowed mode. When a game is run in borderless windowed mode, the game does not have full control of the display– that overarching control is given back to the Desktop Window Manager (DWM).
see also https://learn.microsoft.com/en-us/windows/win32/direct3ddxgi...
If an application framebuffer is full screen and in a compatible pixel format the compositor can do "direct scan out" where the compositor sends the framebuffer directly to the crtc instead of compositing first. I know that wlroots supports that. I'm not sure how much performance it saves to be honest.
And I'll second that most commercial games do go through XWayland, though it depends on what games you like!
1) dpi scaling is in use 2) external displays are used via usb-c/displayport 3) the nvidia memory clock has been scaled down by power management
that said there have been fantastic improvements overall to both stability and response times since kde6 and nvidia-open have stabilized and i haven't noticed any of it in a long time now. it really is pretty great these days.
Wine/proton would need to support it, XWayland would need to support it (Wine/Proton are one major step away from native Wayland support: Vulkan), and finally the compositor would need to support it. Gnome is about the worst compositor that you could be testing any of this stuff on, they are comically hostile towards ideas not their own. The chances of this ever working on Gnome are near zero. KDE is pretty good for support, Hyprland seems to be trying to support every under the sun.
D3D12 dropped support for exclusive fullscreen, and I don't think headsets even go through DXGI but their own swap chain APIs. Why do VR games on Linux need the equivalent from the Wayland compositor?
Games may show it as an option if the engine itself can support other GPU APIs and the devs working on the options menu don't know or care that this particular settings combination is misleading.
[1]: https://learn.microsoft.com/en-us/windows/win32/direct3d12/s...
GNOME has supported direct scanout for fullscreen apps for a while and drm-lease was implemented not too long ago either.
I have experienced issues with my mouse input locking up/jumping - especially noticeable when I'm playing a shooter - but I'm pretty sure that's just my poor ancient mouse giving out.
The Amiga could do it in 1985. Linux has realtime scheduler classes (i.e. SCHED_FIFO and SCHED_RR). These days, even PREEMPT_RT.
Some Wayland composers are already leveraging hardware cursors when possible.
Why is it so hard? What's stopping them from accomplishing this?
I haven't tried any games that use a cursor with Wayland yet so I don't know if it would have an impact there.
I think it has to do with whether or not the game in question is reading the mouse device directly (e.g. through SSL) or via the compositor. If it's reading the device directly it stands to reason that there would be less latency.
Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter), first to shoot wins, and their total latencies are identical Gaussians with a standard deviation of 100ms. Then a 6.5ms reduction in latency is worth an additional 2.5% chance of winning the trade. Maybe you won't notice this on a moment by moment basis, but take statistics and its impact should be measurable.
In ELO terms a 2.5% gain in win rate is around a 10 point increase (simplifying by assuming that single Gaussian is the entire game). That's small, but if you were a hardcore player and all it took to raise your ELO by 10 points was using a better monitor/mouse/OS... why not? Doing that is cheap compared to the time investment required to improve your ELO another 10 points with practice (unless you're just starting).
Also, I think you'd be surprised what people can perceive in a context where they are practiced. Speed runners hit frame perfect tricks in 60FPS games. That's not reaction time but it does intimately involve consistent control latency between practice and execution.
> Suppose two players notice each other at the same time (e.g. as would naturally happen when walking around a corner in a shooter)
This is not true for third person games. Depending on a left sided or right sided peek and your angle or approach, players see asymmetrically.
For example, Fortnite is a right side peek game. Peeking right is safer than peeking left as less of your body is exposed before your camera turns the corner.
I believe distance also plays a part in the angles.
> “I was working with Larry Mullen, Jr., on one of the U2 albums,” Eno told me. “ ‘All That You Don’t Leave Behind,’ or whatever it’s called.” Mullen was playing drums over a recording of the band and a click track—a computer-generated beat that was meant to keep all the overdubbed parts in synch. In this case, however, Mullen thought that the click track was slightly off: it was a fraction of a beat behind the rest of the band. “I said, ‘No, that can’t be so, Larry,’ ” Eno recalled. “ ‘We’ve all worked to that track, so it must be right.’ But he said, ‘Sorry, I just can’t play to it.’ ”
> Eno eventually adjusted the click to Mullen’s satisfaction, but he was just humoring him. It was only later, after the drummer had left, that Eno checked the original track again and realized that Mullen was right: the click was off by six milliseconds. “The thing is,” Eno told me, “when we were adjusting it I once had it two milliseconds to the wrong side of the beat, and he said, ‘No, you’ve got to come back a bit.’ Which I think is absolutely staggering.”
I didn’t think it was. But it is. I promise!
It’s not necessarily a reaction-time game-winning thing. It’s a feel.
With virtual instruments, my experience is that when you get down to ~3ms you don’t notice the latency anymore… but!, when you go below 3ms, it starts feeling more physically real.
That has not been proven in the article. Input handling follows different paths for full screen games.
It would be more of a problem the other way around, if we had to resort to Wayland to get low latency. I think most of us using Linux for gaming and casual stuff are happy to stick to X11 for now and the foreseeable future. It has good support in software, its failure modes are well-documented, and it doesn't add one more layer to the pile of complexity that desktop linux already is; at least, not one us users have to consciously keep in mind as a common source of issues.
https://github.com/ValveSoftware/csgo-osx-linux/issues/3856
From outside it's hard to tell if it's truly protocol differences or just the age of the implementations on X11, but when Wayland came out every project has claimed improvements over the old X11 stack. Also, from the early Wayland days presentations bashed the protocol as something that couldn't be fixed without a rework that was not going to happen due to the dead weight of backwards compatibility and awful older hardware.
As a user applications running on Wayland have consistently improved on how nice things feel if you don't miss your latency deadlines. It's easy to perceive on apps, and straight out obvious in games.
So much for the whining. I do not know what to do. I could record this behaviour, but it somehow needs to show the timestamp of when i intend a behaviour and the time, that behaviour becomes actualized.
I used to be quick with all the hotkeys but i do make so many mistakes now, because the system is simply not responsive and all my instincts await instant action.
Won’t get you end to end latency, but should be able to trace the input events through chrome, and show the context swaps of all the Wayland/dbus stuff.
EDIT: looks like Linux system tracking is more work, but possible https://perfetto.dev/docs/quickstart/linux-tracing
It'd be interesting to see a compositor that can speak Wayland, yet performs immediate, region-clipped draws of all windows like X11 conventionally does.
I might get to run some benchmarks myself next week.
https://jnsn.dev/posts/frametime/ with a follow-up in https://jnsn.dev/posts/fastisslow/ or if you just want to see the code without my jabbering: https://github.com/DelusionalLogic/Frametime
OTOH the gnome developers have been talking about implementing triple buffering in the compositor. Tripple buffering is only needed when your renderer can't complete a frame faster than the scanout of a frame. Given that any modern GPU should be able to composite hundreds of frames per second this make me think something about Wayland or the gnome compositor in particular isn't designed as well as it could be.
Some sort of double buffering? To ensure it's consistently one frame, it would make sense to test on a slower computer via the same monitor.
There was some discussion on HN in 2017 regarding latency differences: https://news.ycombinator.com/item?id=15748146
As mentioned in those comments, it could be from the compositor
Possibly because the best contributors don’t care about GUI anyway.
Otherwise, it's actually far more advanced than Wayland, and it had essential modern features implemented years ago, including HDR, VRR, and others.
Why hasn't it been ported to Linux? Probably NiH, probably it's Google's child.
What would be required to make surfaceflinger a practical Linux desktop compositor is merely some translation layer, equivalent to XWayland, that supports hardware acceleration. Such things have been written, but not open source and they never got traction.
The idea has been toyed with for over a decade: https://news.ycombinator.com/item?id=5317638
It runs on Android/Linux, not GNU/Linux. I would be surprised if there wasn't at least a bit of work needed to handle things like bionic vs glibc, and you'd really want it to act "normal" by being installed to FHS paths instead of Android's weird filesystem layout. All doable, I expect, but it would be a port.
[1] https://developer.nvidia.com/nvidia-latency-display-analysis...
I wonder if there is an issue with high frame rate support in Gnome (or possibly wayland?). It seems like they are actively still working on it:
https://9to5linux.com/gnome-47-3-improves-frame-rate-for-mon...
[1]:https://www.phoronix.com/news/GNOME-Still-Dropping-Latency
The issue is screen-sharing in MS Teams. Does anyone know if that works now?
Since I use Artix, I expect a potentially painful switch and don't really want to have to then do it all again to switch back immediately.
Or petition Microsoft to enable the Wayland-enable bits in their electron build. Electron has supported screen sharing on Wayland for soon 5 years.
Figuring out why there's increased latency is a job for software tooling, but I think this guy's experiment is one of the best ways to measure what users care about: The time it takes for an input (such as mouse movement) to result in a change (such as cursor movement).
Note that this doesn't mean that the Wayland protocol itself is the reason for the higher latency. It may be Gnome's implementation (testing with a wlroots compositor might shed light on this). It may be differences in default configuration options. It may be that Wayland and X11 start up different services, and the Wayland helper processes increase load on the machine. But I seriously doubt the reason for the difference in latency was because the same hardware was used throughout the experiment.
Using a camera allows the methodology to he identical between Wayland and X, while I don't know how to listen for mouse movements from software in a way that wouldn't introduce its own problems. What if the photons are emitted from the screen after the same number of milliseconds across X and Wayland, but Mutter is a few milliseconds slower about notifying my measurement application? Conversely, what if Mutter has more real latency than X, but the latency is introduced after the stage where my measurement application sits?
The variables you mention are identical between the Wayland test and the X test. It is admittedly a challenge for reproducibility, but doesn't affect these results.
Would like to see a test with gnome and the wireless mouse removed from the equation.
USB mice update at 1 kHz.
https://lobste.rs/s/oxtwre/hard_numbers_wayland_vs_x11_input...
> X11 has no cursor vsync and tears. Wayland compositors use atomic cursor updates synchronized with vsync.
(Hi, author here by the way! ... Don't worry, that disabling hardware cursors thing was at least one OS re-install ago)
Mutter definitely throttles cursor updates on Wayland, too, which will contribute slightly to latency even with a hardware cursor. In general, with Wayland, the compositor is responsible for cursor updates, and I'm not sure which other ones throttle. But that would be where the difference comes from when using hardware cursors.
I think the difference mort96 is seeing is the cursor update throttling. If it updates at the same rate of the refresh rate then it's a crapshoot where in that interval it hits relative to vsync, with a worst case additional latency of the frame rate. X11 updates the cursor whenever it moves, so on scanout it's always where you expect it to be, even if the motion was almost immediately before vsync.
I should mention that in the past there's been problematic interactions on amdgpu with atomic updates of the cursor and display planes. This resulted in either one being starved from changing, causing stuttering in one or the other depending how it was handled. IIRC, that's why the throttle is there. You could try MUTTER_DEBUG_ENABLE_ATOMIC_KMS=0 to see if they only added the throttle to the atomic code path.
It's all software.
It has worse performance on almost all metrics. Heavy fragmentation. Very limited and incomplete API measured in functionality but still extremely complicated to work with as developer. Extremely slow development. The only thing that it has over X11 is better funding (for whatever reason).
Mixed DPI works perfectly fine on X11 if the toolkit supports it. The xrandr protocol provides all the necessary dpi information. GNOME simply chose not to support it on X11.
Reading some of this thread has made me feel like I'm going insane. I have a wildly different experience of both X and Wayland from a lot of people in this thread it seems.