The Linux graphics stack from X to Wayland (2011)
arstechnica.com
arstechnica.com
[1] Part 1: https://lwn.net/Articles/188060/
[2] Part 2: https://lwn.net/Articles/188073/
[3] Jim Gettys: https://en.wikipedia.org/wiki/Jim_Gettys
[4] The (Re)Architecture of the X Window System (July 2004 Ottawa Linux Symposium): https://keithp.com/~keithp/talks/xarch_ols2004/xarch-ols2004...
[5] Blog: https://gettys.wordpress.com/
[6] Bufferbloat: https://en.wikipedia.org/wiki/Bufferbloat
[7] Bufferbloat project: https://www.bufferbloat.net/projects/
It is however relieving to see how much progress has been done on the graphics stack since (open source drivers, DRI3) and the Wayland side (especially with libinput, which I think didn't exist at the time of this article).
Anyway, Wayland is almost there, guys!
Really? Most distros still use X11 by default 5 years later. That's pretty bad progress in my book.
I mean, this is more of a upstream-domain issue than a user-domain issue.
Do you think 5 years is bad because the developers just didn't like, invest enough free labor in the project to satisfy your arbitrary requirements, or something? Did they not work hard enough? Was the job actually easier than they made it seem? Would replacing it in 3 years have been acceptable in your "book" of requirements? Inquiring minds want to know what your book says.
I ask this seriously, even if it may come off as slightly annoyed. I think that's reasonable levels of annoyance. Your reply comes off as extremely ungrateful, IMO. I ask that you seriously, sincerely think about why you believe this is "bad progress". Do you have serious reason to suggest they have messed up somewhere, that they have failed to deliver on promises, or do you just "think" that it shouldn't have taken that long and thus, it is bad? Or do you believe X never needed replacing (and why do you think that, vs the Wayland developers, most of whom are prior X developers)? "Thinking something" you can't verify isn't the same thing as actual knowledge.
People who work on open source and free projects invest real time, labor, and life into them, much of it thankless work, on projects like Wayland -- even if the projects themselves are good and that "thankless" work vital. Even if they take years to realize. Why? There are a lot of reasons, really. But none of them involve wanting to have people judge them with no context, by arbitrary standards of how good they are as software developers, or something, because they think it "took too long". That's the kind of shit people make fun of managers for saying!
Please note I'm not affiliated with the Wayland project in any way, but I have my share of experience with doing maintenance on large open source projects. When people say things like this, it comes off as ungrateful, as if I haven't given enough free labor to satisfy you or your desires. Maybe I should have worked 80 hours a week instead to deliver it to you "on time". That might have satisfied the requirements in your "book", I suppose? Or maybe I should have just not written the software at all, and left you on your own, whatever the cost of doing that is.
I've never in my life seen this "tearing" that people keep complaining about. OTOH I use X's network transparency daily. I'm sure several years from now Wayland will be able to replicate the use cases I get now from X, at which point I'll probably switch. So, I wish the best to the people working on it, even if I scratch my head over their intentions.
I'm not sure exactly how X works here. I think as long as you have a visible window (1x1 pixel or more), you can get all keyboard input.
This may or may not be central to your point, but I wanted to have it out there to better inform about how Wayland improves upon X :)
http://security.stackexchange.com/questions/119410/why-shoul...
Admittedly I haven't studied this in detail, so maybe there are some fatal aspects in Wayland that would make my ideas infeasible, but I hope that one day we could have more secure desktop privilege authorization.
Not knowing exactly what Text Expander is, I'm guessing it would probably need access to a specific protocol for controlling accessibility features. The Wayland server would need to be configured to allow this app access to that protocol somehow, for example by way of a server-initiated popup window that asks you to allow or deny this access.
Does that explain it?
Nor should it. That kind of sandboxing should be done as a wrapper around a particular program, not in the display server. A nested X server that sanitizes the protocol wouldn't be particularly difficult. Also, if your aren't fully sandboxing your potentially hostile application, a keylogger is the least of your problems. The display server isn't going to protect you from actual local attacks and privilege escalations; instead, it will only provide a false sense of security.
Furthermore, doing this in the display server itself is shortsighted and unfortunately typical of the freedesktop.org development style that refuses to consider use cases outside their limited personal experience. I use the ability to send synthetic keyboard and mouse events to arbitrary X clients regularly. Occasionally I use the ability to keylog to work around poorly written software. If Wayland refuses to support these features, it isn't compatible with my software.
Security is important, but simply removing features isn't the solution. Gnome/KDE may be popular, but many of us have software that uses X directly. Some of it written before those "desktop environments" even existed.
Nested X servers are a pain, if you've used one - the things you're used to, like functional copy/paste, performant operation, tend to disappear and make this a non-workable solution.
Furthermore, if you look at the release not of (e.g.) firejail, a wrapper around a local program, you'll notice that indeed a lot of the revisions are trying to avoid X bugs that let applications act as key-loggers etc. People have tried, and it turns out to be difficult.
It seem that the same people that believe we need to secure our display server against malicious clients almost inevitably want some sort google play store like experience where people release software for linux in some sort of universal way wherein distros would have no input rather than repo concept that is traditional now.
I feel like they haven't thought this through and will leave us on net substantially less secure.
Here's one data point: It has caused endless troubles on a digital signage project I worked on a couple of years ago. Having a proper solution then would save us tons of hours trying various workarounds to minimize the effect (while at the same time having decent performance).
On the other hand, the last time I (successfully) used X's network transparency was back in '98 on a couple of Sun workstations (when I had access -- most of the time it was VT220 instead :) Since then, every time I did try it, I ended up regretting it since the bandwidth and the latency were too poor for any sane usage (VNC ended up working way better).
Which is not to say networked X is not valuable -- just that it's misleading to think you're (or myself, for that matter) a typical user.
It's a bit like Mapbox: you've got a database full of semantic geometry state that applications can poke (the X server), and then you've got a fronting daemon that renders, serves, and caches graphical "tiles" created by querying that geometry (the VNC server), along with exposing a protocol through which clients can easily poke at the database by referring to it in terms of the tile-server's screen coordinates.
Also, other people might say it's clumsy, but personally I really enjoy the architecture of containerized GUI apps: you've got xvfb running in the container as an X server, and then x11vnc or the like also running in there, talking to xvfb, serving VNC "tiles" and receiving key and pointer inputs over a socket. It's like the OSI network layers: X11 is the application layer, VNC is the presentation layer.
VNC does single window clients which interact with the primary and secondary selections on the display server now?
I agree, in my experience VNC has given better performance than X through the standard network transparency feature.
If you have the need for a remote desktop setup in the future, I'd recommend checking out NoMachine. It's closed-source, but from what I've seen so far it offers better performance than VNC.
So the point of Wayland is to have a clean and simple foundation, with no obvious security vulnerabilities, no overhead, and no kludgey workarounds or ancient APIs.
As far as the user permissions, fair enough, though I personally think hardware access should require root, for the most part.
Apple and Microsoft have tossed out their window managers more than once also. X is older than anything they've had.
I do, yes, which is why I avoid things like PAM and all the various FOO-kits (sadly Slackware has since succumbed to consolekit, which is why I've been moving to OpenBSD). I want to know that anything I do as my user without su or sudo or doas (depending) will not have an impact outside of my home directory.
(Though to be clear, I think a display server could conceivably be justified for a dedicated non-root user -- fbterm seems like a good model here: getty hands over the VT to a user who has write access to the framebuffer.)
Well, writing into the home directory requires hardware access, too. Where do you draw the line?
In the similar way that the kernel's VFS provides authorized access to storage hardware, the *kits (or nowadays systemd-logind) provide authorized access to video/audio hardware.
It requires access to a filesystem, which may or may not translate to writing to a physical disk, depending on the setup.
Yeah, there's a lot to dislike about how X works. It's old, and it shows. But it has had man-centuries of work to get it where it is now; a de novo solution will necessarily need similar levels of work to get to a similar point.
What could radically improve a next-generation X replacement would be to use a higher-level language than C, which would remove entire categories of errors as possibilities. I'm thinking either OCaml or Common Lisp could potentially be excellent choices, for very different reasons.
But replacing one large, old, battle-tested C project with a large, new, untested C project isn't going to usher in a happy world of simplicity and security.
That work does not need to be directly in the second system. If the hard lessoned were learned in X by making mistakes much effort can be prevented by not making those mistakes and use something closer to the finished design of the first system... which is exactly what Wayland is.
That doesn't mean that it's not a net-positive to develop and start using it, of course.
Maybe they were great at the time. If you don't have enough memory to store every pixel of your screen, only storing properties of some basic geometric shapes seems to be a nice solution. If you have plenty of memory and want to control every pixel this becomes rather useless and even a burden.
This reasoning falls down in two ways: sometimes changes need to be reverted (and there's no reason to repeat those mistakes) and sometimes changes make further changes harder (and these changes can be done smarter in the new project).
> a de novo solution will necessarily need similar levels of work to get to a similar point.
No, X contains lots of cruft and hacks.
Yes, that's a feature that some of us use. Any replacement needs to support that ability, or it isn't a functional replacement.
> without compositing you need to re-render the window when it moves into the screen
Some of that is prevented by setting the save-unders flag. More can be prevented if toolkits bothered to draw using X instead of re-sending client-rendered pixmaps on every update (a lot of the people complaining about X are not actually using X for most of their work).
However, isn't the whole point of Wayland is that it's a compositing-only server? It's simply forcing the choice to enable compositing. Enabling the Composite extension in xorg fixed the redraw problem years ago. This isn't a Wayland-only feature.
> X is really just a middle-man that doesn't do much
That's only true for a small set of X clients.
> No, X contains lots of cruft and hacks.
A lot of that "cruft" is backwards compatibility that doesn't interfere with modern applications and is needed to run older software. Choosing to break 20+ years of software by removing these features is not an acceptable solution.
More importantly, a lot of the problems with X are actually problems with the specific implementation of X known as "X.org". A lot of confusion happens when people confuse X (a protocol and specification for display servers and client interface) with an implementation of X ("XFree86", "X.org", others).
No it doesn't. There is no need for any client without any permissions or privileges to be able to snoop on other processes. Why not let users decide which clients can snoop?
> That's only true for a small set of X clients.
What does X do?
> More can be prevented if toolkits bothered to draw using X instead of re-sending client-rendered pixmaps on every update (a lot of the people complaining about X are not actually using X for most of their work).
No, that is how X works. I found out this when experimenting with creating my own toolkit. The Composite extension fixes this.
> Choosing to break 20+ years of software by removing these features is not an acceptable solution.
It is acceptable. Most people don't use 20 year old software. XWayland can be used to keep backwards compatibility.
This is exactly what I'm afraid of. Don't get me wrong, I go outside for a smoke every morning when I find out I have to as much as look at X code that day (fortunately, it doesn't happen too often), but you know what, X11 works. It has, furthermore, worked reliably for many, many years, on more architectures than I can name, running more applications than anyone can remember, and has reliably solved problems that no one even conceived of thirty years ago when X11 was written. For something that "barely holds together", it works admirably well for the end user, and the nowadays-hated extensibility and policy-but-not-mechanism have allowed it to remain a very useful solution.
Wayland, in contrast, is universally applauded, but largely runs Gnome and a bunch of infotainment systems for automotive applications.
X11 has also not always been so reliable, and frankly, I doubt Wayland will take much less time to become reliable. It took about 10 years for X11 to stop being a pain in the ass. I'm looking forward to using Wayland every day. In 2025 or so.
Also, at the risk of looking like someone who wears his tinfoil hat to bed, it's worth considering that the two major improvements are largely relevant only for multimedia-super-heavy stuff and staying relatively safe from a legal standpoint while running 3rd party, untrusted (cough usually closed source cough) applications on a system that you manufacture and sell (oh, and a lot of people who want to move every application in the browser, but that could be solved by just sandboxing the browser). Before jumping into the PR boat and proclaiming the end of The Dreadful X11 Insecurity, it may be worth following the money trail.
Wayland is still being rolled out, it hasn't even managed to be the default compositor on any major Linux distros yet (though I heard that Fedora may choose to default to Wayland for its next release). Furthermore, KDE devs are clearly working on Wayland support, so it's not limited to one DE.
> "Also, at the risk of looking like someone who wears his tinfoil hat to bed, it's worth considering that the two major improvements are largely relevant only for multimedia-super-heavy stuff and staying relatively safe from a legal standpoint while running 3rd party, untrusted (cough usually closed source cough) applications on a system that you manufacture and sell"
What are you talking about? There's no closed source bias with Wayland.
This is true on desktop systems. It's already running fine and dandy on many embedded systems, and it has been for a fairly long time. I have at least four or five boards in my office that are running it, for production purposes, i.e. real-life devices.
Certainly it's going to see more adoption, including by KDE. But at the moment, its support in the desktop world is pretty much limited to Gnome and, uh, that i3-like compositor, I forgot its name.
> What are you talking about? There's no closed source bias with Wayland.
With Wayland itself, no, certainly.
But one of the things that intrigues me is that somehow people would have you think that the open source community -- historically, one of the most security-conscious -- has managed to turn a blind eye to this elephant in the room, a frickin' walking keylogger, throughout the troubled 1990s and 2000s. You think we just somehow missed this disaster waiting to happen, and it took us 20 years to come with a contingency plan?
It is not the only factor, nor the dominating one, but one of the significant reasons why some companies like Wayland is that it saves them a lot of headaches, including (potential) legal ones involved in selling equipment that has to run third party/OEM/crapware programs. Users of these systems may, and often will, input sensible stuff on their devices. You don't want every company who publishes an application to the app store to be able to read everything.
Truth is, this has simply not been a major concern so far. It is becoming.
I don't mean to say it's a bad thing, and like I said in my previous post, I certainly can't wait for when X11 will become history, but I think we need to keep a close eye on what's happening. It's important to understand that a need for device, software and user data control is a major driving factor behind much of today's technology, and it's worth keeping an eye on.
I'm not sure which i3-like compositor has Wayland support, but I believe Enlightenment supports Wayland.
> "But one of the things that intrigues me is that somehow people would have you think that the open source community -- historically, one of the most security-conscious -- has managed to turn a blind eye to this elephant in the room, a frickin' walking keylogger, throughout the troubled 1990s and 2000s. You think we just somehow missed this disaster waiting to happen, and it took us 20 years to come with a contingency plan?"
A lot of Linux servers run without X installed, you don't need GUIs to administer a Linux server. Seeing as Linux first found major popularity as a server OS, the security risks of X weren't quite as major as they may seem.
Also, whilst X has its issues, there were more fundamental issues that took precedence in the earlier years of Linux. I remember when I started playing with Linux in the early 2000s one of the main issues was drivers. If you've ever had the pleasure of trying to get Windows WiFi drivers working with ndiswrapper, you'll know what I mean.
Lastly, it does appear that there were proposals to improve/replace X11 over the years, but proposals are easy. I'd say the growth of Wayland was a mix of the right people at the right time. We're still waiting on something similar to emerge for the audio stack (though I'm hoping the work to improve the VR experience will be the catalyst that helps to drive this forward).
X11 was, if not the most common, at least one of the most common workstation interface throughout the 1990s, when Linux, even on servers, was not as major a player as it is today, and most high-grade workstations ran a Unix variant. There were exceptions (some systems ran NeWS) but, by and large, if you did any important (so usually confidential) work back then, it was quite likely you would do it on something running X11. The risk was as major as it is today.
The i3-like compositor is called sway: https://github.com/SirCmpwn/sway/
As an i3 user, I'm pretty happy to see it so far along in its development.
---Alex
Edit: fixed link
1. Compared to X, the codebase for Wayland is small. This isn't just because Wayland hasn't had enough time to accumulate cruft, it's also because of the design of Wayland. Wayland makes use of a lot of the underlying kernel infrastructure and streamlines the functions it provides. To give you an idea of how the infrastructure of Wayland differs from X:
https://wayland.freedesktop.org/architecture.html
2. Wayland may not be as old as X, but it's not a new project either. Development of Wayland started in 2008, and has been a freedesktop.org project since 2010.
3. Nobody claims Wayland will be perfect, it just has to be better than X to be a worthy successor. Second-system thinking works out fine when the problem space is well understood, and the issues of X appear to be understood just fine.
Also regarding the parent comment, wayland has been tested in production for many years now in lots of embeded systems. If you have a car with a Infotainment system from 2013 or newer running linux it most likely uses wayland. It currently works on the open source intel, amd and nvidia and multiple arm drivers, so the major drivers missing are just the proprietary nvidia and AMD drivers (both of which are being actively worked on).
That's not true. There is a lot that went into X that is no longer in use, and doesn't need to be faithfully recreated; e.g. basically almost all rendering (font and otherwise) has moved from server to client. At least in the past (Latest sources I studied were of X10 ... showing my age), that was a significant part of X, with a lot of work going into it - and it does not need to be redone.
The way I read the previous post, with its mention of "second-system thinking", is that X is currently just about good enough for just about everything, and that while 90% of the necessary features will fit cleanly into Wayland, it's the last 10% that will introduce 90% of the ugly complexity, and take man-centuries to realize the need for, and then produce.
It's not only about things done wrong but also about X11 not fitting very well into todays hardware environment. Today a lot of the X11 protocol (w/o extensions) is either things done really bad or things simply unused most of the time because today nobody uses bitmap fonts and everybody renders into pixel buffers anyway but it has to be implemented because the protocol demands it. The implementation being old and barely maintainable combined with protocol being horribly outdated seems to be a good reason to start from scratch. Wayland as far a I know is also a much simpler concept.
> What could radically improve a next-generation X replacement would be to use a higher-level language than C, which would remove entire categories of errors as possibilities. I'm thinking either OCaml or Common Lisp could potentially be excellent choices, for very different reasons.
Sure C is bad but I don't think languages focusing heavily on a functional programming style are a good choice for this type of software. You probably also need this low level access C provides at some points. I would tend to Rust here as it was designed to also fulfill such needs.
> But replacing one large, old, battle-tested C project with a large, new, untested C project isn't going to usher in a happy world of simplicity and security.
I don't have any numbers but the security track record of this "old, battle tested code" seems not to be the best.
That's an assumption that is only true if you are careful to only use Gnome/KDE style, recently made applications. Backwards compatability is a good thing, and most of those features are necessary for running older software, and they don't interfere with modern software.
> nobody uses bitmap fonts
I use bitmap fonts for almost everything that isn't firefox and ps/pdf rendering.
$ xlsfonts -1 | wc -l
16264
> everybody renders into pixel buffers anyway
GTK+ and Qt render into pixmaps. Other toolkits vary.
> because the protocol demands it
...because backwards compatibility demands it. Maybe you only run recent software, but that isn't always an option.
> protocol being horribly outdated
Fortunately X11 has an extension mechanism which allowed new protocols to be added at runtime, removing the need to use older protocols. This is needed if compatibility is to be maintained. Unfortunately, the fd.o culture seems to think it's fine to remove working code without regard to the people that do use it.
> Wayland as far a I know is also a much simpler concept.
It's simpler because it's feature incomplete. These features still need to exist, and all Wayland is doing is pushing those problems to other areas where they are someone else's problem. In some cases (like use cases outside their experience), Wayland developers like to pretend these problems don't exist.
> I would tend to Rust here
I completely agree that Rust is probably a good choice, as it was designed specifically to avoid many common types of bugs. C is good, but Rust may be the first language that can actually replace it.
[1] Such as Terminus[2], Dina[3]
[2] http://terminus-font.sourceforge.net/
[3] https://www.donationcoder.com/Software/Jibz/Dina/
[4] http://www.antigrain.com/research/font_rasterization/index.h...
Same here, it is the one feature I need. Then again I have seen 'tearing', but only when using nouveau, I still have hopes of nouveau, I really wish NVIDA would help out more than they have been with nouveau.
Knowing what I know now, I would not have bought a MB with integrated NVIDIA graphics. My NVIDIA Chip will soon be out of support, so I will be faced with either buying a new graphic card or living with nouveau.
Wayland is at lower lever than X and it's 'security by default' creates problem so replicating the use case you get from X will take time and many synchronisation/interoperability efforts.
Is there a good write up of this or do I need to gather bits and pieces?
Something like this?
https://blog.gtk.org/2016/06/15/drawing-in-gtk/
The API will likely change considerably in Gtk4, more here:
Fedora 24 continues to improve support for the next-generation display server, Wayland. Wayland is still not the default for Fedora 24, but it worked well enough that I would not be surprised to see it the default option for Fedora 25. Just know there are still some quirks, like the inability to reliably get screenshots working, which is a deal breaker for me. [0] More details in [1].
I'm sure it's available in most other big distros, too.
[0] http://arstechnica.com/information-technology/2016/08/fedora... [1] https://blogs.gnome.org/mclasen/2016/03/04/why-wayland-anywa...
It also was available in Fedora 23. I've used it daily, only with minor hickups.
These hicksups include:
- Gparted not working without some command-line initialization, etc.
- Remmina remote desktop not working at all, unless built from source.
All in all, for a guy like me, pretty easy to work around. The tools which needed tweaking wouldn't be the ones "your grandma" would be using anyway, so impact on a typical user would probably be low.
It's not all the way there, but it's definitely ready for the "I use my laptop for web-browsing and email"-crowd.
https://github.com/rhinstaller/blivet-gui
http://www-rhstorage.rhcloud.com/blog/vtrefny/blivet-gui-20
Not just for servers, but quite useful on them is Cockpit which leverages storaged and libblockdev, providing a good chunk of storage configuration via web UI, among other things. https://github.com/cockpit-project/cockpit
The default DE is still gnome with X11. In fedora 25 (that is next release) they intend to flip the switch to Wayland though.
That'll be the real test for how desktop ready it really is.
Progress is steady though. For example, KDE's Neon project [0] turned Wayland on as default [1] in developer edition.
Wayland will slowly crawl into distributions near you.
[1] https://blog.martin-graesslin.com/blog/2016/09/kde-neon-devu...
One of those things that you really can't understand why they sink money into it :(
You're wrong. That's Unity 8. And it's not the default DE (although you can apt-get install it).
I was looking forward to trying it out, because the Intel integrated graphics (HD4000) has bad tearing problems with application such as Firefox.
That, combined with PulseAudio problems may force me to re-install 14.04. Though I need to keep using Win10 this week because of a CAD program anyway.
FYI, the article is a bit dated. Currently Ubuntu is working on their own display server called Mir [0].