[1] Yes, I know there are exceptions.
X separates the thing that paints the screen (the X server) from the things that computes (the X client).
Your idea or pushing pixels means that you need a virtual screen on the machine where programs runs and you want to copy this virtual screen to your screen. Do you imagine the complexity of window management with such a setup ? Have you already launched many VNC clients at the same time ?
Some people claim it depends on adequately configuring X, which may be true, but it's much easier to configure VNC over SSH to have an acceptable performance
On Windows or OSX though... If you were going to install X11 why not skip it and just install VNC?
VNC is no alternative for running individual applications remotely. Remote desktop != remote X.
Still, most of the time, running only one application is not the usage model. Or it's a simple thing that doesn't need X, just ssh