Another VPS user here. Linode and Digital Ocean offer great guides to get started. I've basically moved two domains off of the formerly free Google Apps accounts onto my own servers. Overall things have gone well. I like being able to set my own file attachment size limits.
As others have said, a 'clean' IP seems quite important. I've had emails disappear into Google's spam filters despite working previously, as reported by many here[0].
Was tempted by Virtualmin but went with the manual config in the end. Am receiving multiple domains on one server, so it has been a learning curve. Not convinced I've 'learned' a great deal particularly. At least a slightly clearer understanding of the moving parts involved. That's Postfix, Dovecot and Spamassassin, btw. All served over SSL/TLS only, i.e. only IMAPS is enabled in Dovecot.
Tempted to start encrypting all mail as it arrives on the server with GPG, as per Mike Cardwell's advice[1], although I'd have to use mailgate[2,3] with Postfix instead of Mike's Perl script for Exim. I'm also not sure how that would work with search, spam and such. Perhaps it's better to delete mail regularly, saving any files and details needed locally.
Had a slight edge-case issue arise recently - my phone battery died and I didn't have my machine so my mail was inaccessible until home. I've refrained from enabling webmail and think I'll keep it that way for now. Just make sure I have 'a device' with the credentials saved on it on my person when required.
--
[0] https://news.ycombinator.com/item?id=9150927
[1] https://grepular.com/Automatically_Encrypting_all_Incoming_E...
[2] http://andsk.se/tag/gpg-mailgate/
[3] https://github.com/uakfdotb/gpg-mailgate