But why should I care about their perspective? When we judge people, we judge them based on our own system of values (morality, ethics etc - call it whatever you like). They are basically axioms that we take for granted, and that provide a starting point for the logical reasoning process. I don't see why societies should be any different in that regard.

So, yes, every time you see a statement along the lines of "society X is better than Y", it comes with an implied "Based on my values, ...". But precisely because it is always there, there's no point spelling it out every time.