You're right - an adversary with physical access to machines running Vault can steal keys from it. An HSM is designed to survive at least temporary physical access by an adversary. And if you have management infrastructure that lets admins access the servers running Vault, then they can exfiltrate the keys (an HSM can be configured to prevent this). There exist threats that HSMs will stop that Vault simply won't stop.

But they are both systems intended to provide secure storage of key material with policy-based access to that key material. As an SSH CA, you can configure Vault to sign SSH pubkeys but never divulge the key material. Depending on your threat model, it might get you what you want for this use-case, but you should definitely be aware of the limitations of Vault's security model.