Skin color is not a good proxy for genetic relatedness because it's only one of the many different traits that make up a human.
You would surely agree that two humans with different skin colors but all else equal are more closely related than two with the same skin color but nothing else in common.
And it turns out that is exactly the case. Africa has the most genetic variation since it's the origin of humans so a black African can very well be much more related to a white European than to another black African.
https://en.wikipedia.org/wiki/Human_genetic_variation
In computer science terms it would be trying to do a dimension reduction by only naively comparing the first coordinate. That would work only if skin color is the principal component of genetic diversity but it's pretty unlikely and insane to declare that a priori.