Doesn't disabling USB cut you off from most of the hardware on a Raspberry Pi?
Come to think of it, doesn't the SD card slot sit off of the USB bus?
Come to think of it, doesn't the SD card slot sit off of the USB bus?
It doesn't.
This can be done with (for instance) u-boot.