Of course, there are availability concerns with that, which you can mitigate with onsite redundancy (dual power supply machines fed separately, etc.), and multi-site replication.

The big issue is how to get keys back into the system. I suggest having some onsite tamper-resistant component and some typed in remotely component.

The annoying thing is there is no viable low-end solution for this yet. It makes sense once you're dropping $100-150k of equipment per site, but for 2 servers per site, it's a huge pain and lots of overhead.