There are a lot of downsides.. or rather hassles to running your own mailserver, this is true, but as someone who has run their own mailserver since 2001 I wouldn't have it any other way.

Email is one of the last modes of communication we have on the Internet that can still fully work w/o any big company controlling it and I think that is something worth preserving.