Hey, nice post! Quite a detailed write-up and an impressively optimized video pipeline. I'm curious how much end-to-end latency you experience on average?
On newer hardware (Geforce GTX 900+ series on the server side, recent Intel HD Graphics on the client) our system only adds 8-10 ms. On older rigs it might be somewhere around 20. Of course the network latency will go on top of that.