Race as we culturally define it (which varies significantly over time and location) doesn't respresent differences in fact, but rather the things that are most obvious to our perception - e.g., skin color. There's no coincidence that what we see most easily - skin color - is somehow a strong indicator.
It's also a misunderstanding of statistics to think that differences will apply to individuals. These aren't two completely separate distributions, but rather distributions that overlap in different ways for different factors that impact outcomes in complicated an often unknown ways. Science and genetics are not that simple.
> That would not happen in the ideal world
Our world is far from that, sadly. What has caused more damage in history than racism - wars, massacres, slavery, oppression (including mass incarceration), endless lifetimes and generations lost to neglect? There have been no wars, slavery, or massacres due to anti-racism.