I did something similar for my macbook with uxplay: https://github.com/FDH2/UxPlay
I ran it in console directly rendering to framebuffer and it has hardware-accelerated video decoding. On Raspberry pi 3, hooked to lan and my macbook on wifi the lag is ok. It's about the same as ssh over slow cellular. I mainly run my terminals watching the logs/builds etc. on that display.