I completely agree!

I'm running an email server for myself since 1998 and in addition some low traffic email servers for a few small companies that I administrate and we have had no delivery problems so far, not even with Google, although I have heard from other people that run their own email servers, that Google occasionally puts delivered emails in the Spam folder.

I always keep an eye if some of our IPs may appear on any of the well known abuse lists, but so far that never has happened. I would guess if once there's coming spam from your mail server IP, it is negatively branded forever.