Sigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually during boot. This way we're protected against many types of hosting provider hacks.
I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part.
But I'm not a security expert. Maybe I've overlooked something.
Properly hard problem, as you've pointed out.
I strongly doubt that:
Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices).
As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard.
I still think that it mostly defends against someone stealing/cloning the physical disks though.
Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.
I suppose the sanest middle ground is to wait for the dust of an event like this to settle (or have someone investigate, in case of "just" an unexpected reboot) -- then wipe the server(s) in question, reinstall (hoping the install media is ok) -- and importing the disks/backup.
I still think it is little gain for a lot of work.
Basically if your server software is secure, you've avoided data compromise due to a reboot into single user mode or similar -- but you have to assume (for the sake of guaranteeing the safety of the encrypted data) -- that attackers have a copy of the data at rest (which they can get if the can boot into a recovery image etc.
So now you have to make sure that you can boot a safe environment in which to import the data to, without leaking your key(s).
All the while, it is rather likely there are a few vulnerabilities in your software stack that all this never did mitigate.
I'm not saying don't do it -- I'm in favour of doing it for the (relative) peace of mind that no that will get lost/out as the servers are recycled -- I'm not sure it makes much sense to avoid hackers -- the online threat seems to dominate for most cases I can think of.
It's relatively easy on Linux, where /boot is usually not encrypted.
This is the same methodology I use, and I think about these attacks a lot. There is no good/cheap way to verify a remote execution environment right now with commodity hardware. :/
It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it.
If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks.
If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.