As others already noted, having more than one key is a good idea anyway. If you are really serious, use ssh certificates so you don’t have to update every server with new keys. Just sign them with the root CA.
You can never actually grab it or access it for backing up, so it shouldn’t be your only way of accessing a server, there should be another authorised key that you do have access to.
Because they are generally not very configurable (their design goal is to be secure and so the less complexity the better) it’s fairly common for them to just not directly support any specific cryptographic protocol.
Given that, what you can choose to do instead is have the hsm generate a key for you, and then you use that key to wrap your specific secret - say an ssh key - then you decrypt it when you need it which requires user authentication through the hsm - use the raw key and then clear it from memory.
But if the only record of the external key is wrapped by the hsm, if the hsm loses that decryption key then you’ve lost access to the ssh (or whatever) key as well.