Desktop environments on Linux basically just draw into a full-screen X window, don't they? (And then the programs you run are children of that root window.) I wonder how hard it would be to hack up XQuartz to support that...
The main issue with XQuartz nowadays is that it doesn't support Retina displays in windowed mode. Everything runs pixel-doubled by default.
There are only two options: hack Virgil 3D or hack XQuartz. You cannot "just" run XQuartz and solve problems. And I choose hacking Virgil 3D because it should have less communication overhead and work with Wayland.