Compositors are an optional component. I have never run one myself since they don't add anything of value. So any time someone says "the X server only talks to the compositor" I'm like "lol what compositor".
But even if you have one, it is still wrong. You have to understand that GUI applications are a LOT more than just low-level rendering. So even if you didn't need it for that, there's so much more going on that the server still plays its role in like coordinating copy/paste, drag+drop, hotkeys, notifications, etc., etc., etc.
And it isn't true that everything goes through the middle man. A graphics-heavy X application actually tends to work very similarly to a Wayland application (not coincidentally - as people love to point out, a bunch of the Wayland devs worked on the gpu rendering parts of X before): you get a direct render context, write to a buffer, then inform it to draw it. See more https://keithp.com/blogs/dri3_extension/
Mixed DPI is fully supported by the system, but may be buggy in applications. See here http://wok.oblomov.eu/tecnologia/mixed-dpi-x11/ you can get information and handle it client side for maximum quality, or do auto-scaling for easy support bridges. KDE and Qt manage to do it. So much for "unfixable".