You can do this with even less fiddling just by getting a KVM that supports video. There are reasonably priced ones that can even do 4K 60Hz. This also means you don't have to deal with monitors that don't implement input switching via DDC/CI (thanks LG).