> I use the same monitor as well. Until recently, I had the 15" version of that laptop, and now I have the latest 15" Macbook Pro. Both use 2560x1440 on the external monitor even though it's capable of 3840x2160, because that's what OS X defaults to, and every time I try increasing the resolution, I only make it a few minutes before I can't stand how terrible the experience is and end up switching back.
I would suggest to try using different DisplayPort cable.
OSX defaults to "best" resolution for the given display. In case of this monitor, it is 3840x2160, but - it will tell you is 1920x1080 and use it in HiDPI mode. Like this: http://imgur.com/a/infej
Once you change resolution, it will remember, what you changed it to on that specific display, and will use it next time you connect to it.
> I'm saying that native Mac applications look really bad on OS X when set to 3840x2160. The UI and text don't scale consistently, and that's true both of the system-level UI and of application UI.
In my experience, they scale perfectly. The ones that are non-HiDPI aware, are scaled by the system to the proper size (and thus are pixelated).
XQuartz is another matter. It should be taken behind the barn and shot long time ago. Running X11 apps is better done using a linux vm.
However, what I don't understand, why you have wrong scale. XQuartz is a not DPI-aware app, thus it is scaled by the system.