A very important issue with Key Vault is: what to do when the Hardware Security Module dies? All electronics fail or stop working at some point. How do you make backups of keys that were on the HSM?
YubiHSM back in the day, I recall reading, was designed so that you'd want two HSMs, one generates random secrets, the other stores the secrets using keys that never leave the device, if I recall correctly. And the reason it needed two is that the generator would leak parts of its keys with the random data it produced, I think, and so to securely store them, you needed a second device with key generating turned off. I could be out to lunch here, never bought a YubiHSM nor do I have experience with corporate ones. My point, is that there are different uses for HSMs, and it's easy enough to have an insecure use of HSMs, even as simple as generating secrets and storing secrets on the same device.
As to what to do if the key is lost, I suppose it's time to re-issue. :) The goal is to not make too many backups: keeping a key secret is more important than ensuring the key is widely available, right? So it's a balance....
Security of key material is all about procedures. With a private CA I helped to setup, we used a quorum based authorization scheme, and the collection of smart cards was distributed among different reporting lines to make collusion between employees difficult.