I did some work at a large saas company where HSM as a big part of what we did. In the early days, we paid for full HSMs that we managed and eventually had to make a decision on whether or not we would used managed HSM and HSM-like services like KMS across various cloud providers.

We did an analysis across all the HSM hardware vendors and found, unsurprisingly in hindsight, that all of these hardware vendors had all the same awful security practices as every other enterprise hardware vendor, and each had vulnerabilities that leaked private key material.

The conclusion was that cloud providers fronting managed HSM had more to lose than the hardware vendors did, and would be more likely to patch and address these kinds of issues.

What surprised me was my own reaction, I thought for sure managing our own HSM hardware had to be a better guarantee over the key material, but like many things, it turned out to be unscalable security theater.