I think I see what you are saying, an I agree that practically speaking, we have to have some common ground to make things work.
So at some point there is also a line that has to be drawn. The Holocaust is one of those lines, and I pray that I would had the strength to stand up to my peers and say, no this isn't right,and go to whatever lengths I thought I could do to make change. In a lot of ways, I think the civil rights movement was a lot of chest beating and I think that actually helped.
It's not really clear to me that "most prosperous happy places on earth tend to have social contracts that look out for groups such as women, homosexuals, etc." Certainly, when you look at economic prosperity and I don't know how you'd measure happiness.