Ah, thanks for the info :).
I'll admit I haven't had to mess around with X for...probably 5 years? The last I remember, you'd run an X server on your machine, then your window manager would connect to it on the loopback device.
The point stands, though, that "getting rid of localhost" is incredibly naive.