Can’t you just run X11 on the Mac and Linux in a vm?
The main issue with XQuartz nowadays is that it doesn't support Retina displays in windowed mode. Everything runs pixel-doubled by default.
There are only two options: hack Virgil 3D or hack XQuartz. You cannot "just" run XQuartz and solve problems. And I choose hacking Virgil 3D because it should have less communication overhead and work with Wayland.
/edit: oh, and yeah, if you patch xrandr not to return, I suspect that the resolution change will persist on switching between macOS and the unix desktop, though I haven't tested that yet.