Calling Xorg garbage is naive.

Even if you only use terminals, and you have to have a framebuffer that supports that native resolution of your screen, you'll find that Xorg is nearly always more performant.

If you have issues with X11's security model that's a different story, I'd suggest trying http://wayland.freedesktop.org/