sloccount says xserver is around 370kloc. This isn't counting the drivers outside of the generic KMS support (which, practically speaking, is what you're probably running on anything remotely modern), but ati+amdgpu+nouveau are another 102k, and intel is another 186k (partly because it embeds a copy of the software renderer for dumb reasons), so you're still not quite at two-thirds of an MLOC. The size isn't so much the issue, it's the design. The xfree86 ddx seriously believes that you have one video card driving one monitor with one keyboard and one mouse, and anything beyond that only works well because we sank a ton of effort into it. (It used to have code to disable interrupts! From userspace! What could go right.)
It sounds a little like you think I'm being given these priorities as marching orders from above. That's maybe a bit insulting? I suspect I'd have the same opinion at this point regardless of my employment history, namely, that X is a tremendously successful project whose core design and reference implementation do not reflect how computers work anymore, in ways that make it painful to develop and maintain as the system display server. Frankly that was probably true in 2005 when I started on it, but at that point it was also the only thing that had credible video drivers at all.
It's not that Red Hat is trying to lock anybody into a particular desktop out of some nefarious political agenda, it's that we don't have the resources to do things twice. If we want to deliver the best desktop experience we can then choose your own adventure among Gnome and KDE and XFCE and twm and i3 and whatever else simply is not going to be an efficient use of our headcount. And we're reaching that point with display services too, trying to support both X and Wayland increasingly means writing the same feature twice, sometimes with radically different approaches, and trying to implement them _at_ _all_ under X11 is more and more intractable. What would you have us do? What we're trying to build now is an Xwayland that keeps your X apps working as well as they ever did, under a hardware display server model that makes things like high-dpi and HDR and AirPlay and RDP and GPU offload straightforward, or at least feasible, instead of excruciating. You can like that or not, I guess.
In all seriousness, if someone wants to take the reigns of stewardship over the xfree86 code then please, by all means, come forward and claim your prize. But I can't justify spending my time on that anymore, and given the interval since the last major release, apparently nobody else can either.