Only the public key should be stored on the virtual mail server, since all it's doing is encrypting incoming mail. It's your local email client (e.g. Thunderbird) that then downloads the encrypted emails to your local machine and decrypts them with your private key. Signing and outgoing encryption are also done locally. At that point, all that someone raiding your VPS gets is a bunch of encrypted messages, which good encryption is supposed to make worthless, and your public key, which they should already have (along with everyone else).

Sure, your local machine could also be raided and the keys grabbed from that, but that's not a problem with using a VPS so much as a problem with your local setup.