The point is, although your example is possible, the probability of this happening is pretty close to zero.
This becomes more likely if you include genocides that may have quipped out large swaths of someones descendants.
You also have rather insular groups where a few thousand people may be largely segregated from the rest the world. That's not going to be particularly stable over hundreds of years, but you only need 1.