Most KVMs that fail at 4K get the EDID wrong. They either send their own fixed one (often maxing out at 1080p or 4K30), or they reconnect the display on every switch, so the OS thinks it's a new monitor and resets your scaling. Look for one with EDID emulation.

For high refresh 4K you also need HDMI 2.1 or DisplayPort with DSC support, and a lot of "4K" KVMs don't have it.

Also, 1080p divides evenly into 4K, so turn on integer scaling in the Nvidia or AMD settings and 1080p will look sharp instead of blurry.