I don't expect there to be much debate about conditions like "they gave you their email address on your website and your email has an unsubscribe link."

It obviously wouldn't be a perfect system, but neither is any reputation formula. And the consequences of being wrongfully blacklisted are incredibly mild: I'd just skip our follow-up emails, or maybe require blacklisted users to double opt-in.