The latency would probably make it a shitty experience. I got a displayport switch when I started working from home this year and the thing works so well I forget it exists. There is no latency at all and it automatically switches inputs based on which one is active so I never have to touch the thing.