When I ran a cloud provider, we used a hardware security module on storage systems which had public/private key pair which could be used to decrypt the header on the disk which contained a copy of the symmetric cipher key. Each header had the symmetric cipher key encrypted with multiple different public keys, including a fallback whose private key only existed in on paper in a vault.

Each system could reach out to other storage systems to ask them to use their private key to decrypt the header (for example if their hardware security module had failed), but in some configurations this would require an operator to intervene to enter in passphrase to unlock the hardware security module to authorize the action.

This means that:

1. The symmetric cipher key could always be recovered, even from paper backup

2. Having physical access to a disk or any set of disks did not allow you to read the data

3. Having physical access to a disk and a hardware security module did not allow you to read the disk (unless you knew the passphrase, which was always present, and user set)

4. Having physical access to disks, servers, and hardware security module may not allow you to read the data (in the more secure configuration where passphrases were not cached on disk -- but this meant that rebooting required an operator to manually enter the password at boot)

5. The set of valid public keys could be changed frequently (this was indeed automated and only the set of currently active hardware security modules could decrypt the current disk header)

Of course you could always short circuit this by making a copy of the symmetric key when you did have access (e.g., `dmsetup table --showkeys`) but without putting some more hardware in the fast/hot path that was unavoidable. The symmetric key could not easily be changed, without rewriting the entire disk (though since it was a storage system designed to accommodate multiple failures, this wasn't that hard, but we didn't do it automatically)

The hardware security modules we used were FIPS 140-2 validated and physically connected to the racks (though it would be possible to cut them away). It would also be possible to spy on the APDUs sent to the modules to capture the decrypted data, since there was not mutual authentication (it was in the works though).

I thought any of these providers always encrypted with their own key whether you encrypted on top of that or not. But I guess not.

Yet another reason to just stick with the big providers. They seem to have pretty serious compliance needs and encrypt no matter what. (Yet you should still encrypt your assets, but I believe they all encrypt their volumes anyway and have pretty strict policies on how to dispose this kind of hardware)

That's what Dropbear is supposed to do, it can be included in the initramfs of a server: https://matt.ucc.asn.au/dropbear/dropbear.html

Disclaimer: I still didn't try it myself yet, but it's on my (too long) list of things to do.

I assume they don't mean to refer to the physical building, but to the organisation responsible for the building and its physical security. Anyway that's a thing for auditors to have fun with (so you say you're hosting provider is certified, can you show me their certificate?) as you're not supposed to lean on their website information anyway.

I don't think many of the competitors in this space (eg Linode and DigitalOocean) own their datacenters either.

As Linode put it: Linode as an entity is not ISO27001 and we rely on the data centers we colocate with to obtain and maintain these certifications

Or DigitalOcean in 2017: Our FRA1 facility is ISO9001:2008, ISO27001:2005, and ISO22301:2012 certified.

(DigitalOcean later got their own certification)