That's true, but to do it securely isn't as simple as one would like to believe. Setting up "DKIM/SPF/etc." simply provides auth/verify security on send, but I'm more specifically referring to the receiver side security.
If you're really properly securing your mail server, it would likely be isolated behind a firewall and only have a LAN ip of some kind and utilize UUCP for transport to another LAN machine that does not have WAN access, and then, only allow POP3/IMAP access to machines in the LAN or connected to the LAN via VPN tunnel. Finally, you would want to setup a backup system of some kind for this machine to periodically backup via rsync when the inotify/fswatch file modification triggers.
Next, you'd have a separate SMTP machine. For things like critical deliverability, you can't rely on SMTP to 'retry' albeit it's how they are supposed to act, so it would make sense to have multiple SMTP machines across multiple different backbones in different physical locations with backup power and the like with different MX priorities set.
The initial configuration and running a mail server are incredibly easy.
It's running it securely that increases the difficulty on order of magnitude (because you essentially have to setup a proper security protocol across multiple machines (a network) - defense in depth).
That said, it's easily doable if you're already running complex infrastructure. Hopefully, you're getting paid for your time and costs for doing so.
If not, then I hope you need to rely upon the protection of needing a home-invasion warrant vs a simple-subpoena since a machine at your home can't really just be 'subpoenad' while a machine at some datacenter business can. This of course assumes you're even running the machine at home because if you're doing all this on some VM the value of doing so diminishes ever so quickly.
EDIT: Just to be clear, you can't simply rely on fail2ban and some other on-machine script / snort / daemon / kernel feature to protect you. There are bugs in software/systems and 0days are very real (as well as the market places for them).