I don't think it's about labeling people as being good or bad, but more about not letting things stand.
If you encounter attitudes from people that are hostile towards you, you can either speak up and allow them to correct themselves (or double-down) or be silent and give your tacit approval.