Modern Desktop Linux is very fragile because everything has to be configured and managed automatically. You haven't been able to startx without a lot of reconfiguring for years.

Ideally you want a full Desktop environment, but at a minimum you need a session manager (for auto-starting everything else, communicating with the * kits, * buses, * agents, and * managers for device integration, etc . . .), a settings daemon (stores settings for applications that care to use it, might be more than one of these - Xfce has two or three at the moment), a desktop manager (like a WM but for desktops), a window manager, a compositor (for special effects, might be part of the WM), plus miscellaneous bars, launchers, and plugins for same.

It's perfectly possible to split all this up, but then you have to make all the parts compatible. Most distros want more integration, not less.