You generally do one (or both) of two things with hardware security devices:
* You buy multiple devices and configure your systems to honor both, as a backup plan.
* You back up your keys or their artifacts to paper or a small drive and keep that somewhere safe.
If you do neither of those things, and you lose the hardware key, you are either (a) boned or (b) using an insecure system where the key is just theater.