I had that setup working back in the day with a Voodoo1 which had the pass-trough construction to take over full screen: only on Linux you could read the completed frames and render them in an X window. It was dog slow though.
While transfers have improved a lot the amount data needed for 4K60 with HDR is about 18Gbps. There's no way that doesn't have an impact.