To take an empirical equivalent, imagine if someone gave you two lists of people, one of whom was "tall", and the other was "short". And you measured the tall and short people, and there were people in the "short" group who were taller than average for the "tall" group, and vice versa. Their height as compared by means and standard deviations didn't vary substantially.
At that point, your assumption should be that the labeling and grouping was wrong. So too for grouping people by skin color or many other arbitrary groupings like ethnic groups and national origin.
That being said, there are groupings of people by genetic traits and things like that do matter - the classic example being broad African-American susceptibility to certain genetic-influenced illnesses like diabetes and heart disease. These are markers that more recent African immigrants don't show, and are likely a result of strong genetic and epigenetic selection for certain kinds of robustness as a result of the historical conditions they were subject to, i.e. slavery, discrimination, and poverty.