The science of not getting flagged is easy when you're not sending large volumes of untrusted mail; it only gets complicated if you start hosting mail for "customers" or let your system forward mail unfiltered into gmail/yahoo.
Here's my hit list of universal things to configure:
* Start with an IP with good or neutral reputation, non-residential, its nearly impossible to fix an IP that has been burned by a spammer. (Network)
* Valid reverse dns for your IP matching your mailhost forward dns (DNS)
* Valid SPF record; -all (DNS)
* Valid DKIM; with sufficiently sized key (DNS+Config)
* Valid DMARC; start with p=none to test and move to p=reject once you're configured (DNS)
* ARC if you or your users will ever possibly forward mail (Config)
* Don't get your messages flagged as spam anywhere ever, filter outbound mail even if its just you. All it takes is one piece of malware and a saved password and you'll have to get a new IP. (Config)
* Don't configure services behind your mail server with example domains that you don't control ~ I get so much mis-configured test mail from people who think its cute to use my domain as an example in their practice lab. It all gets reported as spam or bounces and then their smart host bounce rate goes up. (Config)
* Test for open relay; only relay for authenticated users. (Config)
* Use strong authentication, preferably with certificates or MFA. (Config)
* Secure everything; IMAP/SMTP/POP are old AF make sure you're requiring STARTTLS and setup MTA-STS to prevent downgrade attacks and enforce encryption in transit. Use a real certificate from Lets Encrypt don't self-sign. (DNS+http+Config)
* fail2ban your auth, you're going to get so much driveby password spraying and credential stuffing; I fail2ban block entire subnets at a time with iptables actions. I also have a bunch of "poison pill" rules for weird stuff I see in my logs eg block anyone who tries to auth with the NTLM hash for 'password'. (Config)
* Don't bother with BIMI at home, you can't get a blue check mark without deep pockets and a trademark (vmc) and most platforms only show logos that have a matching vmc. (DNS+https+config)
* DMARC reporting and TLS-RPT reporting are a pain to manage but are helpful troubleshooting deliverability be prepared to read some XML reports or setup a stack to parse them as they arrive (DNS + Config + https)
* setup the SMTP Submission port (587), so many networks block port 25 outbound and its the right way for clients to connect. (Config)
* configure BACKUPS, don't skip this step, encrypted restic backups to s3 or backblaze b2 is cheap and easy. (config)
* track your configs in git, don't commit secrets. (config)
* configure a free blacklist monitor on mxtoolbox for your domain(s) (config)
If you do those things you'll be in a pretty good spot, you could probably paste that list/this post into your agent and vibe up solid mailserver.
For me keeping the spam and phishing out is a bigger hassle than deliverability issues. rspamd does a pretty good job of keeping it manageable.
I do all of those things and with all of that setup the only place I ever run into issues with with users on AT&T's residential broadband mail servers. AT&T appears to block you if you're not known to them and they have a short memory. If you don't have regular correspondence with AT&T users they will block you after a bit. I'm a fairly low volume sender so I end up blocked every other time I try to send to AT&T by no fault of my own. I've talked most of those friends off of AT&Ts free email and on to ProtonMail at this point.