4k @32 bit color is ~3.5Gbit/sec at 60 fps or 1.6Gbit/sec for 30fps.
Modern video cards and CPUs are fast. With no optimizations I can screen grab a fully rendered X11 4k x 5 screen in ~100ms on i7-4790 CPU from a GTX 1060 ( did it yesterday when I discovered that Looking Glass does not support Linux guest). Some people that I know who are in the graphics business claim that using MIT shared memory extension they can do nearly 1000 fps for a single 4k screen.
Iperf (TCP) with no network optimization and no shared memory a-la Looking Glass between host and guest gets 2.5Gbit/sec.
Why not just do a PCI pass through to a virtualized ( or dedicated ) graphics card to guest, run an X server on it and stream raw frames from guest to host? It surely can do at least 30fps on a modern workstation or over a 10G network with no frame loss.