Google's keyservers also don't have single employee control. The ones used for encrypting gmail behind the scenes, and other Google services. This is just table stakes for any large system.
This is in contrast to something like Azure Key Vault, which is HSM backed. A court order should not be sufficient to compel Microsoft to turn over a key from Azure Key Vault; it should be impossible for them (or for nCipher) to do so.
HSA is an Amazon term for a PC with an HSM inside. The data-at-rest might be protected by the HSM (full disk crypto with a dongle used to decrypt at boot), but the actual keys get decrypted into the host PC's RAM, and further customer-accessible calculations happen in the PC CPU.
Anyone who can tamper with the PC can read the keys!
There are two risks this exposes you to:
1) Someone goes into the datacenter and physically attacks the HSA.
2) Someone legally compels the owner of the HSA to subvert the HSA.
I'm not as worried about #1 (these are equinix tier-4 datacenters; someone rolling in with some M4s and a bulldozer and such is great for Hollywood. Insider threat probably still exists with the HSA even though normal operation is two-employee, though.) I'm incredibly worried about #2, since the bar for #2 is hella low for emails older than 6mo.
I believe ECPA older-than-6mo would be sufficient to compel the email KMS key as mere instrumentality, so even a the fairly low bar today of warrant wouldn't be required.
3) Exploit programmatic access or side-channel attacks on the data.
If the server can decrypt the data and this is driven by code on the box, then you're in a DRM-like situation trying to hide data from a program that has legitimate access.
As you alluded to earlier protecting data at rest doesn't protecting during use.
It depends on how you configure the HSMs whether you can extract and decrypt keys.
Why does it matter? They can certainly be compelled to use the HSM to decrypt data, even if they can't extract keys.
If you aren't doing client-side encryption and keeping the keys private, the server has access one way or another.