Sure, but in general the expected value of a HSM is that (especially in FIPS mode) it prevents extraction and accidental leakage of the keys (and maybe at appliance levels has explicit logging).

If you're sure you never do a plaintext backup or anything else that might leak the key, then it may not matter. But it is a decent defense in depth measure for high assurance operations.

I heard a story once about a CA (public? private? not sure!) that had an airgapped root CA in a basement lab, only for an enterprising admin to run an Ethernet cable between their desk and the basement lab to save themselves a few trips... An online HSM may allow effective leakage (by allowing arbitrary signing operations), but hopefully would prohibit export by a crypto user and retain audit logs of all operations to identify scope of the compromise.

At some point, you end up reinventing an HSM from first principals. :-) My 2c.