I used to self-host my email. I don't any more, for exactly this reason. And of course this applies to every other "just do it yourself!" enthusiast appeal. I actually want to delegate it to someone who does it full time.
And the big one: inbound anti-spam.
It's not that issues are frequent, it's just that there's no guarantee they won't be badly timed. There may come a point in your life where (a) you don't have a lot of free time and (b) you badly need a specific email.
I still have the mail domain, I just pay an ISP to handle it for me.