I'm honestly interested in building a better X11, and am willing to contribute both time and money. But first, I'd like to understand why the X11 maintainers deemed Wayland necessary -- like, what am I missing here?
Which problems, exactly, would these be? Why is it impossible (or too difficult or disruptive) to solve these problems through a server extension, or some other backwards-compatible fix? I'm sure the X.org developers have an answer, but I'm not seeing it here.
> it is highly unlikely that the major desktops are going to want to continue on that path going forward.
I don't care what other desktops and toolkits choose to do -- I don't even use a fd.o desktop (hell, I don't even run dbus). If it weren't for needing a Web browser and Zoom, I wouldn't even need a GUI at all.
I'm doing this for myself. I like UNIX a lot, but I really dislike the direction modern Linux desktops are going in. But instead of whining about it online, I'm willing to put in the time and effort to keep things working the way I like.
I'm asking what problems Wayland solves in order to figure out why it's a bad idea for me to take a crack at implementing a simple X server of my own (assuming X.org is indeed going to be deprecated). Like, what is so wrong/broken about the X11 protocol that the X.org server developers are so enthusiastic about abandoning it? Clearly, I must be missing something. I would like to know what that something is.
Side note, I don't get the hate for dbus, it's a rather simplistic message bus, orders of magnitude smaller than the X server. It would be much easier to implement your own dbus daemon for example.
This doesn't call for ditching X11 in my mind.
> For me personally, the real bad issues are things like the core protocol being synchronous, the coordinates being limited to 16-bit, the inherent raciness and insecurity of various things like window properties and server grabs...
Why can't an extension offer a way for clients to establish asynchronous communication channels to the X server? Why can't an extension offer 64-bit coordinates? Why can't an extension offer a way for an authorized program to take care of guarding and serializing access to window properties and orchestrating server grabs? Why do we need to break the world to have these things?!? These may not be trivial undertakings, but I doubt they would take anywhere close to the amount of work required to upgrade every graphical program and toolkit in UNIX-land to use a wholly-different _suite_ of input/video multiplexing systems (which on a given day will be only 95% compatible with one another in expectation).
Also, it's not like Wayland is destined to be less crufty than X11. I wouldn't be surprised at all if all of the complexity in X.org today returns to Wayland compositors by way of a bunch of all-but-required Wayland extensions that get shoe-horned in over the years. So if we're going to be shoe-horning new features into existing systems, we might as well do it on the devil we all know (or perhaps we should solve this once and for all by creating an "X12" protocol in a way that shoe-horning is painless and won't lead us to cruftiness again).
> Side note, I don't get the hate for dbus, it's a rather simplistic message bus, orders of magnitude smaller than the X server. It would be much easier to implement your own dbus daemon for example.
It's not simplistic for what it does, and its developers have a horribly-misguided "put-it-in-the-kernel-because-performance" development ideology that belies a profound lack of understanding of why or how dbus isn't fast enough for their purposes. My biggest turn-off is the fact that it doesn't do anything that I can't already do faster and cheaper with a RAM filesystem of named pipes and UNIX domain sockets.
* Want user/system namespaced paths? Create a directory for each users' endpoints that's separate from other users' endpoints, and have a distinct system/ directory that only authorized users can explore. Leverage filesystem hierarchies and permissions to communicate which endpoints belong to the same service, and to control who can access them.
* Want to register a service endpoint for suspending/shutting-down your laptop? Create a directory under system/ whose group ID is the group of users who are authorized suspend/shutdown, put two "suspend" and "shut-down" named pipes in them, and have a suspend/shutdown daemon just do blocking reads from them. Once a byte arrives on the "suspend" pipe, execute suspend-to-RAM. Once a byte arrives on the "shut-down" pipe, execute shutdown.
* Want to register a service endpoint for sending desktop notifications? Make a "notifications" directory in the user's service endpoints directory, and put a UNIX domain socket in it. Have the notification daemon listen on this socket, and simply pop up a window whenever another program connects to it and sends a properly-structured message (note that that other program must have permission to traverse the service directory to access this UNIX domain socket to do so).
* Want introspection on how to form that message? Have the daemon that implements the endpoint write out a symlink to its documentation in its service directory, which you can just `cat` or `more` to figure out how to talk to the service.
* Want something really elaborate, like sending a video stream? Transfer a file descriptor to the service provider via the UDS and then pipe the audio/video data in that way.
So, yeah -- dbus doesn't need to exist in order for us to have the things it offers.
If you added all those things as X extensions, it would essentially be the same thing as Wayland, because clients that wouldn't use them would still be broken, and every graphical program and toolkit would still be need to be updated to use them.
Your suggestions for dbus would work for some applications but would not really work for other things that a message bus handles like multicast, global message ordering, and resource accounting. Plus GNOME and KDE adopted dbus specifically so they could get away from having to pass around random sockets in folders everywhere. I assume by "put-it-in-the-kernel-because-performance development ideology" you're referring to kdbus, which was an alternate implementation not made by the original dbus developers, and is now a dead project and is not really a thing anymore. Please don't get those things confused. Of course the reason they could do that is because dbus is also just another protocol with a reference implementation, and you could make another implementation that works closer to what you describe and maybe gets 80-90% of the way there depending on some changes in the kernel, for example I saw a hacky dbus fuse filesystem a while ago: https://github.com/sidorares/dbusfs
There's a massive difference between extending the X server and having each window manager implement all the trappings of an X server as a library. Namely:
* X remains the "narrow waist" for video/input multiplexing. GUI "policy" infrastructure -- window managers, panels, notification services, and so on -- remain separate programs, with separate maintainers, to be mixed and matched downstream as needed. Moreover, all these programs keep working.
* By remaining a separate X server program, we keep mechanism and policy cleanly separated. GUI "policy" infrastructure can't impose itself systemically on other GUI "policy" infrastructure, which is a good thing because all these GUI "policy" infrastructure authors tend to think their way is the best way and how dare anyone question it or resist it (see also GNOME). X keeps me and mine safe from their idiocy.
* The barrier-to-entry for creating new "policy" infrastructure remains low, since you can run these programs without coupling them to a particular compositor.
* Changes to X's rendering infrastructure get incrementally deployed. No change in any workflow is required; toolkits and programs opt-in to the new rendering infrastructure as they need to. Programs that don't opt-in keep working until the old code paths get dropped.
* Non-display services of X get preserved, like xprops, xinput, etc. All xclients keep working. If desired, these can be policed through a separate opt-in extension. Existing IPC conventions like ICCCM and NetWM keep working, so all the downstream tools that use them keep working.
> Your suggestions for dbus would work for some applications but would not really work for other things that a message bus handles like multicast, global message ordering, and resource accounting.
Nonsense.
You can multicast messages from one process to many processes via a UNIX domain socket trivially -- just send the damn message to each recipient! It's not like you're going to have 10 million clients, so copying the data isn't going to be that bad (and, the service endpoint can always throttle clients). But, sure, let's suppose the message you're trying to send is gigantic, and you do need to send it to lots of clients. You can just store it as a file (you're doing this anyway if the message is truly that big) and send each client a read-only file descriptor to go and consume it at their own pace. If you're using a file at least, all your clients will hit the same cached pages in the kernel, so you're no longer making N different copies of the data (the kernel will take care of implementing the right caching strategy for you). If you're streaming data, you could simply buffer it to a file and treat the file as a ring-buffer, and still hand out read-only file descriptors to it to downstream clients.
Global message ordering and message dependencies is also easily solved without dbus -- just implement an "ordering" service adapter. The adapter writes its own UDS to the place where its upstream services' UDSs live, and it takes care of marshaling requests and replies to and from the upstream services according to some ordering principle you require. For example, if you have a service for shutdown/suspend, and a service for logout, you could implement a small ordering adapter that prevents messages to shutdown/suspend from being delivered if the user is in the process of logging out. I'd imagine that for a DE, you could simply have a singleton ordering service adapter that determines what services get to be accessed under which circumstances (thereby cleanly separating the task of systems integration from the task of providing the individual service).
Resource accounting is similarly straightforward. Just like the "ordering" service adapter pattern, you can also create a "resource usage" service adapter pattern. For example, you can ensure that the volume increment or decrement requests to your sound daemon arrive at a fixed rate, no matter how many requests come in. As another example, if the service is streaming data, you can use a service adapter to monitor how quickly clients are consuming versus the service producing, and induce back-pressure on the service to hint that it should down-sample if clients are too slow.
Because everything is represented as files, I can do those last two things trivially with shell scripts. No need to take over the init process (cough systemd-logind cough), no need to implement a whole wire format and marshaling library and stub-compiler, no need to create language bindings, etc. Files, directories, named pipes, UNIX domain sockets, and a humble script to set desktop-wide policies on inter-service interactions are more than adequate. But noooooo, we had to build dbus and all of dbus's infrastructure.
I honestly believe the authors of dbus simply lack imagination. Like, we have all this wonderful battle-tested POSIX IPC infrastructure sitting around waiting to be used that they don't even have to maintain, and the kernel makes a fine I/O multiplexer and request broker. Why not use it to its fullest potential? It'll save time and effort, and you won't need any specialized SDKs or tooling to interact with services.
I don't want to say that I think the dbus authors are, well, stupid. If there's something that dbus does that well and truly cannot be done as described above, I'd love to know what it is, and why it justifies all the complexity of re-implementing POSIX IPC analogues in a bespoke system. But I've been writing software for over 20 years, and I've been around the block plenty of times, and this entire project smells like something someone would have written if they simply were not familiar with what their runtime environment could already offer them.
> Plus GNOME and KDE adopted dbus specifically so they could get away from having to pass around random sockets in folders everywhere.
So instead we should just implement worse-performing analogues of most of the POSIX IPC primitives in userspace and pass around service endpoints instead? Come on now.
> you're referring to kdbus, which was an alternate implementation not made by the original dbus developers, and is now a dead project and is not really a thing anymore. Please don't get those things confused.
Thanks for correcting me. I wouldn't want to hate on people for the wrong reasons ;)
------
Anyway, we've been going back and forth for a while. I'm convinced now that Wayland is just an instance of CADT and doesn't solve anything that couldn't have been solved with a less-glamorous but less-effort X extension. But whatever -- the X.org and fd.o developers are free to do whatever they want, etc. etc.
I actually like the X11 model, and wouldn't mind taking a crack at writing a Wayland compositor that simply back-ported all the non-graphical aspects of X as a Wayland extension. Then everything I'm using today could, ostensibly, keep working (and I don't have to care nearly as much what the fd.o folks do going forward).
If you don't care about policies then all those things in X can be a good thing, but if you do care about policies then Wayland could allow for a better design, at least it seems that's what GNOME and KDE are aiming for anyway since their policies are very well established at this point, and they don't really seem to care about breaking ICCCM and other such things.
As for dbus, your solutions would work for some things, but would not have exactly the same semantics as dbus and would come with their own set of issues, and requires building several more infrastructure pieces, some of which you just described. You could build those but it likely wouldn't fit the same use cases as dbus. If you're sending messages that you expect other clients to parse then you still need to agree on a wire format and marshaling library, you can't get around that. If you ask me dbus itself doesn't require much infrastructure at all, you should consider reading the source code for the dbus reference implementation at some point because it's actually pretty small and stable. And I don't understand what you mean by re-implement POSIX IPC analogues, dbus is essentially just a wire format for Unix domain sockets and a message bus that routes the messages, it doesn't re-implement anything. If you want to use dbus from shell scripts, you can use tools like dbus-send and busctl, or you can try to use something like that dbus fuse filesystem -- the nature of dbus makes it map pretty well to that, there's no reason you can't have both a message bus and an easy interface to access from shell scripts.
(Also just another nitpick here, the systemd developers are not the dbus developers, and systemd-logind doesn't take over the init process, that is its own smaller daemon)
If you want to read more, see some comments from the original dbus author:
Before you try and tone-police the above, you should know that it is fd.o that needs to convince me to venerate their software artifacts. People writing more code isn't by itself praiseworthy -- code is a goddamn liability, so it had better have a good reason to exist and (in dbus's case) have a very good reason to be widely depended-on. Just because you happen to like or use someone's code doesn't mean that it is any good.
> And I don't understand what you mean by re-implement POSIX IPC analogues, dbus is essentially just a wire format for Unix domain sockets and a message bus that routes the messages, it doesn't re-implement anything.
I guess if you didn't understand POSIX IPC, you wouldn't see how this sentence is an oxymoron. The kernel itself gives you all the trappings of a message bus for free. You don't need a wholly-separate daemon and wire format spec.
Also, the only people who seem to use dbus's wire format are dbus clients. Even when dbus was new, there were already widely-used and well-understood formats for representing structured data (e.g. ASN.1, typed netstrings, S-expressions) that could have been leveraged to make interacting with the service that much more straightforward. But then again, we're talking about people who wanted to re-invent POSIX IPC, so I guess I shouldn't be surprised they also wanted to impose their own wire format on the world.
> If you don't care about policies then all those things in X can be a good thing
I know better than anyone else on Earth what graphical policies are good for me, so I'm going to take this as your affirmation that X is indeed the right tool for the job for people like me who know what they want out of their computers. I stopped using DEs years ago because I got tired of having to fight them all the time to get them to do the things I needed.
[1] https://security-tracker.debian.org/tracker/source-package/d...
I also still don't see what you mean about dbus, the Linux kernel itself doesn't specify a wire format for arbitrary messages, and doesn't specify all the things that you need to get the complete functionality of a message bus. Maybe you could get that with another operating system that is based around message passing but Linux is not that. The methods you describe could technically be done without a daemon, but they still require a lot of additional code to set up a bunch of files and sockets and enforce ordering, security, etc, which could also contain bugs. You could tell the applications to implement all that themselves or you could put it all in a daemon which is mostly what dbus does anyway, and by doing it in one daemon it totally eliminates a certain class of race conditions and synchronization issues. Again please refer to the comments by the dbus developer that I showed, this conversation is not new and already happened years ago. If you want to store ASN.1 or S-expressions in a d-bus message you can do that pretty easily. And if you really believe that your solution could work then I would encourage you to develop a dbus implementation that works like you describe and then test to see if it works exactly the same and doesn't break existing setups. But I don't think this would really work, you wouldn't really be saving many lines of code, and in particular multicast and service activation would be pretty hard to do in the way dbus does it without a central message bus.
If you don't agree with GNOME or KDE's policies and you want to implement your own IPC then that's great, I support you doing what you need to do, however they chose dbus a long time ago, and currently it's looking like X is not the right tool for them anymore, so you may just have to accept your differences and move on.
Look, I have very strong opinions on what software I consider worth running on my computer, because I've been doing this for a long, long time. I also have very strong opinions on how to go about solving the problems that dbus, udev, systemd, logind, PulseAudo, and the rest of the fd.o middleware purport to solve. I also happen to strongly disagree with how they go about doing it. But, please don't misconstrue this as me believing that their authors don't have a right to create whatever software they see fit. I put my money where my mouth is on this and write code to do the things I want if I can't bear to use the code they publish -- in fact, I have my own binary-compatible udev replacement waiting on stand-by but ready-to-go in case udev comes to hard-depend on systemd [1].
I normally don't share my opinions publicly like this because it causes people like yourself to crawl out of the woodwork and throw well-meaning but unsolicited advice and github links at me to wrappers and adapters for projects that depend on the very software and the very architectural paradigms I'm trying to avoid. When I try to explain this is not what I asked for, it falls on deaf ears. It's a supremely annoying, frustrating experience.
I don't know if you remember, but the only thing I was trying to find out in this entire comment thread is whether or not Wayland solves a problem that was truly impossible to solve with an X extension. I've already got my answer: no, it does not. So, I think I'm done here.
Couldn't find a good article on short notice, but there's a decent video from back in 2013 about it.
I can totally get behind doing a clean re-write of X.org (possibly in a memory-safe language this time around) in order to get rid of legacy cruft that's truly no longer used. They could take the opportunity to refactor the super-popular X extensions like GLX to have better "happy paths" in order to make the overall implementation cleaner and easier to maintain. This could even be done incrementally in order to avoid breaking existing clients.
What I'm struggling to understand is what's so wrong with X11-the-protocol and the popular extensions that ditching everything was considered the best idea? Like, if the X11-to-Wayland transition were happening on the Web, it would be a lot like Google deciding to ditch HTML/CSS/Javascript in favor of something home-grown. Sure, that homegrown thing might actually be better, but it would really leave everyone else in a real lurch now, wouldn't it?
But really, most applications do not deal with Xlib or Xcb _anyway_, they are programmed at the Gui toolkit level. So all that has to be done is add another backend to the few popular toolkits in use. But guess what, Wayland supports both Xcb and Xlib protocols through a virtual X server that transparently translates to Wayland if that's what you have your heart set on.
But I have lost track of what the specific problem is you're actually trying to solve.
I'm trying to fix the problems with X11 that supposedly justify Wayland's existence, because I have reason to believe that fixing/extending X11 would be far less painful and far easier than throwing it all away.
However, no one seems to be able to explain what is unfixable about X11. I assume in good faith that Wayland exists because there is something truly unfixable. I'd like to know what that is.
Note that I'm talking about the protocol here. People here (yourself included) point out that X.org is old and hard to maintain. This may be true, but that is a problem with the reference implementation, not the X11 protocol. Thus it doesn't in my mind justify Wayland's existence (but it does justify writing a new reference implementation).
EDIT: Here's an example -- what if someone wrote an X server that only allowed clients to render via DRI3, and by default prevented programs from receiving keyboard or mouse events intended for other programs? There would be a new protocol extension for setting and querying these blocking policies, so integrators could set more-secure default access controls without breaking compatibility. Isn't that basically what Wayland is aiming for -- client input is isolated and everything graphical happens through off-screen rendering to client-controlled GEM buffers?
> You could do that but such an X server would not really be any practically different from Wayland.
Not quite -- the X server would still provide all the device-independent IPC, input, and screen multiplexing facilities and APIs. Dealing with input isolation could be addressed with an extension.
So I think this answers my question -- Wayland isn't anything special. It sounds like I'd get a lot of mileage out of taking wlroots and adding back in all the device-independent X11 protocols as a Wayland extension. This would basically be the "X server with only DRI3" I described.
In Wayland those tasks have been split out into libraries. The details of the protocol IPC is handled by libwayland, the input is handled by libinput. Screen multiplexing is specific to the compositor and not really something you can farm out to a library, which is the same as composited X where the compositor process takes over the entire screen and handles all the rendering.
It would be interesting if someone combined an X server with a Wayland server like you described, but I don't think it would be useful. A lot of your legacy applications would still be broken, for example no old clients or window managers are rendering using DRI3. If you want to design an extension for client isolation, the problem there isn't that X doesn't have that but that the existing methods don't really work well. My suggestion there would be to talk to any desktop environments to find out what their requirements are, if they haven't already committed to switching to Wayland already. (i.e. GNOME and KDE already have their solution for this in Wayland) It may be that an additional X extension is unnecessary for what the other desktops require.
So? I never said I cared about the portability of low-level rendering software. It's not like anyone cares that Xenocara and the aperture driver only work on OpenBSD, for example.
> Screen multiplexing is specific to the compositor and not really something you can farm out to a library, which is the same as composited X where the compositor process takes over the entire screen and handles all the rendering.
Hold up. Isn't screen multiplexing and compositing exactly what libwayland gives a program the power to do? You'd build and run a compositor (like Sway, or like Kwin), and it fulfills compositing, screen multiplexing, and so on, as well as IPC, window management, hotkeys, screenshots, etc.
At least with X, these were separate programs you could mix and match.
> It would be interesting if someone combined an X server with a Wayland server like you described, but I don't think it would be useful.
I'd find it useful. I don't care if no one else does, since I'm writing this for myself.
> A lot of your legacy applications would still be broken, for example no old clients or window managers are rendering using DRI3.
I'd add the necessary compatibility code for the programs I need to run. I'd add them in a way that, if others wanted to fork my code, they could easily restore their own legacy code paths.
> If you want to design an extension for client isolation, the problem there isn't that X doesn't have that but that the existing methods don't really work well.
Sounds like a problem with the particular extension, not X11.
> My suggestion there would be to talk to any desktop environments to find out what their requirements are,
Don't care. I'm not doing this for them. I don't use any of them, and they're all dead to me at this point. I'm doing this to keep my minimalist X11 window manager and X11 clients, and to satisfy my intellectual curiosity.
Libwayland is a small library carrying the implementation of the wire protocol, and a few other bits like a simple event loop for servers and a library that can load X cursors. The point with that is that you're bringing your own compositing and multiplexing anyway. From there it's optional if implementations want to put in additional features for IPC, window management, hotkeys, screenshots, etc, and they can choose if they want to put that in the server or put it in a separate program. So you can still mix and match on some level anyway, it's not quite the same though.
I had exactly the same idea as you a few years ago to build something like that and I thought it would be useful too, and I thought about it for a while and realized that it doesn't really give you any of the benefits of Wayland or the benefits of X11. The point with wayland is already that it strips the unnecessary bits out and maintains a legacy code path with XWayland, and the point with X is that it's always going to keep the legacy code running anyway, so you don't gain much by combining them. If you have a minimalist window manager that's only a few thousand lines of code, and you want to get the benefits of Wayland, it's much easier to just port that using wlroots or something than it would be to rewrite the whole X server. That's just my experience.
>Sounds like a problem with the particular extension, not X11.
Since this is an issue with Xorg lacking the right extensions it's basically the same thing.
Allow me to qualify: ON LINUX! Sorry if that wasn't blindingly obvious when I said I was totally on-board with a DRI3-only X server.
> The point with that is that you're bringing your own compositing and multiplexing anyway.
I did not have to do this in the X server world. I did not have to worry about inter-compositor compatibility, because there was only one compositor implementation. Wayland has made me have to do extra work for absolutely no gain. This has got to be the fifth time I've explained this on this comment thread. I don't know how much clearer I can be.
> From there it's optional if implementations want to put in additional features for IPC, window management, hotkeys, screenshots, etc, and they can choose if they want to put that in the server or put it in a separate program. So you can still mix and match on some level anyway, it's not quite the same though.
I'm glad you've understood that N different window managers will now have N different ways of doing this, whereas before, N different window managers only had 1 way of doing this. It's nowhere near close to the same -- now in order to do one thing reliably, I have to implement it N different ways.
It's infuriating that none of the Wayland fans seem to see this as a problem. It's almost as if they're the ones who won't be suffering the consequences of their bad architectural decisions!
You do have to bring your own multiplexing in the X world if you were implementing an X compositor. Maybe that's unfortunate to you that the focus changed to X compositors, but Wayland didn't change the fact that this work has to be done by some willing party to get that to work.
I wouldn't call myself a "Wayland fan" but what you are saying isn't really a problem, the different window managers can choose to implement it just one way. They don't have to do it N different ways, of course they will do it differently if they have a valid reason to. From an application developer perspective you shouldn't have to deal with this problem, I'm sorry if you are a toolkit developer and this has caused you pain, but in my experience nearly all of the bits that you would need to have to do a native port to Wayland aren't specific to any window manager.
Not holding my breath. Just because you have a standard doesn't mean all implementations behave the same. In fact, they usually don't, which is the problem.
> but what you are saying isn't really a problem
Thank you for proving my point about being in a position where you don't have to suffer the consequences of your bad architectural decisions.