But I can just pull the plug on any USB 3.0/HDMI/DP cable without any warning to the devices and nothing breaks, why can't a rudimentary switch do the same?
However, the challenges I am referring to above mostly have to do with the steady state operation of the bus, after it has negotiated it's final data rate and is transmitting at full-blast. That's when you run into the transmission problems (eg. radiation of your signals), or your setup just won't allow you to reach the full data rate due to interference.