It really is not. At most it can be tedious, because it has large surface area of protocols and formats. Most of them other than SMTP and message format are optional.

Minimal mail system involves just postfix with config in a bunch of files, delivering mail to a folder on the server, and pretty much everything on top, like SPF,DKIM,DMARC,... is optional.

You can ssh to your server from anywhere and run mutt there, or install a webmail if you're into web stuff. That's pretty much all that's needed for a basic thing. Security wise, there's not much to do wrong in this setup. Mail server is open for reception by default as it should be and submission is only allowed from localhost.

Webmail can be protected via HTTP auth and ssl. As long as your password is secure, noone will get there that way.

Rolling your own crypto is a much bigger can of worms.