Specifically, how -- again, in LINUX-LAND -- a whole bunch of people decided, "nah, we're going to go ahead and break the HELL OUT OF backwards compatibility this time, even though we pretty much never do this."
Look at audio. PulseAudio is mostly pointless and over-engineered. Breaks a lot. You could say the same about ALSA even. FreeBSD meanwhile is still using the OSS API that Linux was on in the late 90s...
The specific issue with OSS was that it was never part of Linux and then with OSS 4 they made it proprietary, which killed it off entirely as far as Linux users are concerned. (FreeBSD cloned it instead.)
No joke, if you remove the new thing stuff magically starts working again.
... except when you have accidentally ended up with half-pipewire, half-pulseaudio setup due to half-forgotten instructions that were no longer applicable.
I have to say, that pipewire under NixOS had been easier to deal with than both pulseaudio and ALSA on crappy internal sound devices (i.e. ones requiring dmix and the like). It's nearly as plug&play as ALSA on "proper" soundcard was (like when I got a Dell Precision to work with that somehow had a Creative Audigy with 256 channel sound, so no need for dmix at all...)
Huh?
But alsa was/is also enormously complicated with a huge library, user mode plugins and whatnot. It reminds me of a common problem that people have where they think API and implementation are one and the same, and that you cannot swap out a different implementation keeping the simple API. People would say you need alsa because it does software mixing, but the lack of software mixing wasn't an OSS API problem, it was a problem with how it was implemented.
The assumption that you could punt software mixing to hardware or deal with limitation of only one program accessing the audio at a time took a hard hit when Intel HDA pretty much decimated presence of such hardware on PCs (AC'97 was much less pervasive for various reasons)
That being said, it's still the same problem.
And I think that's why I find it odd that I haven't heard the argument more plainly stated like:
"Oh look -- right around when Linux became mainstream, it began to get worse in precisely the same ways proprietary stuff was already bad. Probably a result of more business involvement, but how can we counter it."
If it wasn’t designed for that on purpose, I’m pretty sure it’s at least why Red Hat ran so enthusiastically towards it.
Also, it isn't them that are making them the standard. It's independent distributions choosing to use what they produce (and not all of them do, either). Presumably, the maintainers and packagers who make those choices would be aware of these technical considerations, and capable of rejecting Red Hat's tech I'm favor of whatever hoary stack you prefer if it made it harder for them to "keep up." That seems to be something that is perfectly possible to do while still producing a usable distro, and it seems like something limit distributions are quite good at — ignoring what corporate operating systems are doing and forging their own path. Maybe it's because the technologies you are labeling as Red Hat technologies actually offer substantial improvements and push forward the cutting edge of the Linux desktop in a meaningful way, bringing it closer to the capabilities of a modern operating system?
So only interfaces became limited to what's available at that moment and the limited set of Unix software abstractions. Those abstractions are also made extremely use case specific even though there are opportunities to unifiy them like Android did with Binder, Windows did COM and dotnet.
Of course technical difficulties of designing efficient and long-term surviving interfaces also play a role. There are too many hobbyists in the Linux desktop world and too few professionals who also have to work with hobbyists. Many contributions to Linux desktop happen when people are studying and then they leave. This creates disincentives to design long-term systems because they take too long and they are a slog to design and keep up-to-date.
A lot of the user community was fooled into believing that Wayland would make graphics and gaming much better any day now ^2, which never really was based on sound technical arguments.
^1 This was roughly the time when everybody wanted to build new phone or tables operating systems to capitalize on an emerging mobile market. So maintaining stability for the desktop wasn't a priority.
^2 With grotesque nonsense claims about X such that the old drawing API slow apps down even though they are unused, etc. Those "arguments" were endlessly repeated and you can still see this even here, also everybody who ever implemented a GUI application in Linux should realize that this can not be true.
First off, backwards compatibility wasn't actually broken. X11 apps work fine under Xwayland. All the weird bikeshedding that happens in Wayland isn't as important because we have working compatibility bridges and all the old stuff still works.
Second, "don't break userspace" is specifically a Linux kernel policy. The only other organization in FOSS that has such a slavish devotion to backwards compatibility is WINE[0]. Desktop environments are perfectly fine with, at the very least, breaking ABIs, because you can just recompile the ocean. I suspect this is the Free Software equivalent of "firing shots to keep the rent down" - i.e. making the software neighborhood undesirable for proprietary software vendors who will have to deal with these annoying and arguably pointless transitions every couple of years.
The reason why we needed to get off X11 is very simple: X11 is an extremely poor fit for modern hardware. The protocol supports simple drawing commands and image display, that's about it. Modern user interfaces want applications that draw onto GPU layers, composite them, and then present a final image to a compositor to be displayed to the screen. You almost can build that on X11 (modulo some frame tearing), but it's a pain in the ass and requires adopting a lot of extra protocols plus XGL which breaks network transparency[1].
The motto of X11 is "mechanism, not policy". The way this is accomplished is by presenting the entire desktop as a tree structure of windows that any client can mutate. Any widget toolkit can then build the experience it wants on top of that tree structure. The problem is that this also makes writing keyloggers and RATs trivial. X11 doesn't sandbox applications, so even if you lock down a process every other way, they can still do horrible things to other X clients and Xorg won't stop them.
Wayland fixes this by tightly restricting what clients are allowed to do to the desktop. Applications get to present their own windows, sure, but they can't touch other windows unless a specific extension is provided for their use case and the compositor allows the application to use it. In other words, Wayland is "policy, not mechanism". The downside is that now we have to codify all the slightly-different ways each widget toolkit, window manager, and desktop environment has done things under X11. This has resulted in an explosion of extensions, many of which overlap because they were made by different DEs. Wayland can of course create standardized versions of these extensions, but each standardization is an opportunity for bikeshedding.
This leads into my favorite way to tell if an application is Wayland or X11: launch Xeyes. If the eyes track your mouse over the application's windows, it's X11. Wayland doesn't have a protocol for mouse tracking, so Xwayland can't report where the mouse cursor is, unless it's on top of a Wayland window that it's already getting events for - namely, the app you're wondering about.
Ok, I suppose that is a backwards compatibility break.
[0] This also means the most stable UI toolkit on Linux is actually USER.dll.
[1] In fact, I suspect this is why Wayland was so willing to casually toss that out. GPUs and network transparency are allergic to one another.
I believe the question is why Wayland happened so badly. Sure, nVidia shenanigans are contributing to bad fame, but arguably that's not the issue with Wayland. However, Wayland fixes some of X11 design flaws but ignores others - or even introduces new ones.
But, for example, HiDPI is a mess in X11 - for (I assume) pretty obvious reasons of not having HiDPI back in the day. Weirdly, Wayland does nothing to make things right - instead it just gives up on the DPI concept altogether as if physical dimensions simply aren't a thing[1]. While this sort of solves some of the issues X11 had (although, scaling works with X11 too), it's just wrong.
Then there are a bunch of other controversial design decisions (like client-side decorations - as if consistency wasn't a problem already) that are more nuanced.
___
1) Or so I believe - I could be wrong. I just wanted to configure DPI for my monitors hoping that it would help me to have things sized correctly, and have read somewhere that it's not a thing in Wayland and all they have is scale factors.
Because Wayland's base protocol was unable to replace X11, virtually nobody adopted it when it was initially released in 2008. The subsequent 16 years have been taken up by a grueling process of contributors from various desktop environments proposing protocol extensions to make Wayland a workable solution for their users. This has resulted in endless bikeshedding and requires compromise between different stakeholders who have fundamentally different viewpoints on what the Linux desktop should even be (GNOME vs everyone else). As an example, the process for allowing a client to position its own windows has been ongoing for over two years, and has led to multiple proposed protocol extensions with hundreds of comments on each.
The transition would have been much smoother if Wayland had the same "mechanism over policy" philosophy as X11 and allowed for software to easily be ported to it, but long-term that could have caused the same issues that the Xorg maintainers were facing when they created Wayland.
Both of these were shit shows which only look good now to people who weren't around when they were happening.
And systemd was (and still is, I guess) controversial, and there are certainly some design decisions that are questionable - but even in its early days, at the very least it worked for a number of basic scenarios and the majority of issues I've had or heard about was either in its limitations (I remember having issues with journald) or because folks didn't want to change from their preferred rc system as it already worked well for them and systemd was radically different. So, I think, it was less of a shitshow than Wayland. Or maybe I'm just forgetting things - it was a long while ago.
In principle, it is an extensible generic remote buffer management framework. As such, it could be extended indefinitely without every breaking backwards and forwards compatibility and is also a good fit for modern hardware that essentially deals with remote buffers on the GPU. As someone doing HPC programming on GPU, network transparency and GPU are definitely not at all allergic.