Right, but a shadow on dark skin is much lower contrast than a shadow on light skin. The most contrast you are going to find on a dark skinned face is on the eyes. But even then, you can run into problems if, for example, they are wearing sunglasses or you're not close enough to get the focus to lock on the distinction between their eyes.
This might not be a problem in a brightly lit scene, as long as you are using manual white balance (or, well, shooting RAW so you can adjust all that shit after the fact), but as you go to darker locations, you can definitely have an issue.
A possible solution to this would be using infrared depth sensing rather than contrast or phase detection or a laser rangefinding system for autofocusing.