1. TCP streams require stalling when packets are dropped to keep the stream in order
2. X _requires_ by design that commands are performed in order
Which means that using something that can do UDP and manages it's own sequence ordering can do significantly better. This is why things like RDP, PCoIP, etc could do full frame rate HD video 15 years ago and you still can't with X protocol over the network.
Breaking up the screen into small 16x16 chunks or so, encoding on the GPU, and shipping that turns out to be significantly faster.
Especially when you take into account that virtually _nothing_ draws with X using X drawing primitives. It's almost all using Xshm for anything non-trivial.