You could try solutions oriented toward gaming like Parsec or Nvidia Gamestream with Moonlight. They use hardware acceleration for video encoding and may have better latency than RDP.
Moonlight is open-source, you can see in the code yourself as I don't understand most of it. There is also a project that attempt to replace the associated Nvidia Gamestream server but it is quite basic and no longer maintained : https://github.com/loki-47-6F-64/sunshine