The genetic difference between two of the most distantly related humans is actually less than that of a typical ape family. This leads geneticists to estimate that our species went through a bottleneck about 70,000 years ago when there must have been less than 20,000 humans alive.
Furthermore, for specialization to occur usually requires isolation and a whole lot of time for mutation and differential selection to occur. The human race is branching off into different species of human to this day, just give it another 50 or 80 thousand years and there will be multiple species of human.
Humans dominated this planet inside 10 thousand years, in evolutionary time frames that is a blink of an eye.