Regarding your last point: it's getting better. Modern TVs have "game mode", where latency is minimized. The latest LCD panels use embedded Display Port, eDP. The latest eDP standard introduces a frame buffer on the LCD itself, where you can just write deltas and tell it to "swap". I think this is quite similar to what you're proposing, in concept.