This doesn't call for ditching X11 in my mind.
> For me personally, the real bad issues are things like the core protocol being synchronous, the coordinates being limited to 16-bit, the inherent raciness and insecurity of various things like window properties and server grabs...
Why can't an extension offer a way for clients to establish asynchronous communication channels to the X server? Why can't an extension offer 64-bit coordinates? Why can't an extension offer a way for an authorized program to take care of guarding and serializing access to window properties and orchestrating server grabs? Why do we need to break the world to have these things?!? These may not be trivial undertakings, but I doubt they would take anywhere close to the amount of work required to upgrade every graphical program and toolkit in UNIX-land to use a wholly-different _suite_ of input/video multiplexing systems (which on a given day will be only 95% compatible with one another in expectation).
Also, it's not like Wayland is destined to be less crufty than X11. I wouldn't be surprised at all if all of the complexity in X.org today returns to Wayland compositors by way of a bunch of all-but-required Wayland extensions that get shoe-horned in over the years. So if we're going to be shoe-horning new features into existing systems, we might as well do it on the devil we all know (or perhaps we should solve this once and for all by creating an "X12" protocol in a way that shoe-horning is painless and won't lead us to cruftiness again).
> Side note, I don't get the hate for dbus, it's a rather simplistic message bus, orders of magnitude smaller than the X server. It would be much easier to implement your own dbus daemon for example.
It's not simplistic for what it does, and its developers have a horribly-misguided "put-it-in-the-kernel-because-performance" development ideology that belies a profound lack of understanding of why or how dbus isn't fast enough for their purposes. My biggest turn-off is the fact that it doesn't do anything that I can't already do faster and cheaper with a RAM filesystem of named pipes and UNIX domain sockets.
* Want user/system namespaced paths? Create a directory for each users' endpoints that's separate from other users' endpoints, and have a distinct system/ directory that only authorized users can explore. Leverage filesystem hierarchies and permissions to communicate which endpoints belong to the same service, and to control who can access them.
* Want to register a service endpoint for suspending/shutting-down your laptop? Create a directory under system/ whose group ID is the group of users who are authorized suspend/shutdown, put two "suspend" and "shut-down" named pipes in them, and have a suspend/shutdown daemon just do blocking reads from them. Once a byte arrives on the "suspend" pipe, execute suspend-to-RAM. Once a byte arrives on the "shut-down" pipe, execute shutdown.
* Want to register a service endpoint for sending desktop notifications? Make a "notifications" directory in the user's service endpoints directory, and put a UNIX domain socket in it. Have the notification daemon listen on this socket, and simply pop up a window whenever another program connects to it and sends a properly-structured message (note that that other program must have permission to traverse the service directory to access this UNIX domain socket to do so).
* Want introspection on how to form that message? Have the daemon that implements the endpoint write out a symlink to its documentation in its service directory, which you can just `cat` or `more` to figure out how to talk to the service.
* Want something really elaborate, like sending a video stream? Transfer a file descriptor to the service provider via the UDS and then pipe the audio/video data in that way.
So, yeah -- dbus doesn't need to exist in order for us to have the things it offers.