Some of the physical security devices even doesn't support exporting private keys. You ask something (encrypt/decrypt/sign) with a mailbox interface and you get the results. Key is never (and can't be) exposed outside of the secure enclave. Said keys are generated in an isolated RNG inside the key, too. So the system is completely self-contained.
I understand the desire of flexibility and convenience, but this is a tradeoff on a slider. Some designs are full-in on security others are not. We should make choices according to that.