I don't think anyone was claiming that skin tone mattered. Imagine you have a structure supported by two pillars, where one is painted red and the other green and weaknesses in one pillar will make the whole structure fall apart.

No one gives a damn that the pillars are different color. However, it is not politically correct to point out that there are structural weaknesses in the green pillar that need fixing simply because they are different color, so the problem is not addressed.

The pillars could be interpreted as two different communities. What make a pillar weak or strong could be social inequalities, culture, social cohesion, unemployment, etc... Political correctness in the last 40 years has made it impossible to discuss these things in public.

Mentioning the color just serves to describe which of the two pillars has the weaknesses. Nothing else.