Then you've got to set up your domain, and domain headers on your domain host. Oh, DMARC is also another thing.
Then, most ISPs will outright refuse to accept incoming mail from your IP address, since they've basically changed from blacklisting to whitelisting. So you've also got to relay your outgoing mail via your domain host.
And then spam rules. I took the recommended rules from the Debian/Postfix/something-something-sorbs.net website, and I rarely receive email from e.g. eBay, because they've been marked as sending spam. Often happens with gmail addresses, too.
Despite all this... I still run my own mail server, but hotdamn, you're calling this not hard?
EDIT: Oh, and nowdays you've also got to entangle your TLS certificates into the whole process somehow. I managed it, but don't ask me how, I'd need to read up on that.