I used to think like this, and it does seem morally sound at first glance, but it has the big underlying problem of creating an excellent context in which to be a selfish asshole.
Turns out that calling someone on their bullshit can be a perfectly productive thing to do, it not only deals with that specific incident, but also promotes a culture in which it's fine to keep each other accountable.