Consider the 100m dash.
In 2016 Olympics, in the final heat, the slowest man ran 4% faster than the women's world record (which has remained unbroken for 30 years).
Innate differences between the sexes are enormous, even at elite levels.
To put it another way, the difference between a near-average man and a near-average woman (per a normal approximation) could easily go either way. The difference between a top 1% man and a top 1% woman is much more likely to be skewed toward the man.
Seriously, of all things, this one eludes me completely.
People from different climates/backgrounds have different advantages/disadvantages in life, but on average their potential is the same. A Jamaican team could become the best bobsledders in the world if they spent enough time/money/effort on it, though I do admit they are heavily disadvantaged.
https://en.wikipedia.org/wiki/Jamaica_national_bobsleigh_tea...
(Unfortunately, in this case, they lost control during a qualifying run.)
The same argument can be made for VO2 max, armspan, leg stride, etc. Note that I'm not actually advocating for a combined male/female competition nor a split based on genetic potentials.