Philosophically, I agree with you.
The problem is, you cannot do it any more, at least not without significant expenses.
Problem 1: either you choose a provider to handle incoming email for you (basically you want Google Apps, which costs $$$) or you end up with your inbox filled with spam, viruses and bullshit.
Problem 2: Unless your email address ends in Google, Yahoo or one of the other big companies, chances are high that other servers will just dispose of emails you wrote. Especially true if you self-host your server (e.g. an AWS virtual machine, or a rootserver with an IP range of your mass-hoster), because other people's hacked servers will put the entire IP range on public blacklists due to spam originating from these.
Problem 3: you will, if you self-host, have at least three servers to control: your DNS server, your SMTP server and your IMAP server. In addition, you will need to take care of your SSL certificate. All three not easy to do and people will try to hack you all day and night. And you will need at least three machines (two DNS, at distinct providers, and a SMTP/IMAP server). Your DNS must not go down in any case, but your SMTP server isn't as critical because SMTP allows for outages as long as the domain name can be resolved.