Understood. The ideas you mention overlap considerably with the notion of the moral sphere as explained by Robert Kane at the University of Texas at Austin. His books and audiobooks are very accessible and well done.

To summarize, when acting within the moral sphere (which Kane defines as the situation where people are acting in a way that respects other people's rights), we should act to maintain it.

But when it breaks down, we do the best we can to restore it, which might mean acting less ideally toward the people causing breakdown.