That is actually slightly less secure - sometimes people do have physical access to your machines.
For example, the NSA intercepted deliveries of switches and installed their malware on them. Would be easy if there is a physical switch. Not so easy if you need signed firmware (I'm sure the NSA could still do it, but it would definitely be harder).