I have DKIM, SPF and no relaying and the only issue in recent memory was when AT&T started bouncing me for some reason. I emailed postmaster@ saying "hey wats up," and the block was removed a day or so later.
I help with IT for a small non-profit membership org, and the problem isn't the places that throw errors when you send, but the ones that silently accept the message without complaint and route it to /dev/null. We get through fine to Google, but the other big providers are random at best about it.
The downstream MX could be configured to then send an NDR email itself but this lands you on spam lists as "backscattering" will then send spam back to whoever owns the email address the spammer is spoofing so it's generally not done.
"I have no trouble sending to anybody (that I've heard of from recipients" might not be as artfully written as I would like, but that's what that refers to.
Furthermore, blackholing emails like you suggest is generally rare. RFC 5321 (SMTP) states:
"[D]ropping mail without notification of the sender is permitted in practice. However, it is extremely dangerous and violates a long tradition and community expectations that mail is either delivered or returned. If silent message-dropping is misused, it could easily undermine confidence in the reliability of the Internet's mail systems. So silent dropping of messages should be considered only in those cases where there is very high confidence that the messages are seriously fraudulent or otherwise inappropriate."
Yes, maybe rhizome is successfully running his own personal mail server but your anecdote isn't really relevant. We can't learn from your example because the hidden rules of reputation & spam may not affect you but will affect others.
For example, a commenter (lucb1e) argued[0][1] that I was exaggerating the difficulties of reliably sending email but a year later in 2019, he confirmed the same difficulties! [2]
Do you see why we can't reliably extrapolate from personal experience?
[0] https://news.ycombinator.com/item?id=15525505