That's not much of a concern for me. My email server also handles outgoing email as well. I'm fortunate in that I've been running my own server on that domain since 1998. And on the current IP address for probably a decade by now.
Over the years, there have been only two domains where my email had issues being delivered: AOL [1] and Google mail. For AOL, I had to inform AOL that I was responsible for the IP address of my server and promise not to send spam and that I take care of any spams from my IP address, etc. Annoying, but once done, no issues. Google on the other hand ...
I've had a few times where my email was being dropped by Google mail. Annoying, but the issues did seem to clear up on its own after time. I do have a gmail account [2] that I use for testing when I think there are issues but I don't recall the last time I had one. But I have had them.
[1] My dad is still on AOL.
[2] Google's spam catching ability isn't 100% either. I don't use gmail at all for email, so anything I get there is either 1) a mistake because someone is trying to email a person with my name at gmail or 2) spam. It's mostly spam [3]
[3] I just checked my gmail account. 140 new messages since I last cleaned it out in mid-October. All spam. Way to go, Google! (but to be fair, it shunted 726 messages to the spam folder).