The problem is just 1 or 2 or even 10 people washing ashore on an island wouldn't be enough genetic diversity to sustain a population.
IMHO concern that the population wasn't large enough for multiple people to attempt the trick ignore the timescales involved. These events could span tens of thousands of years--longer than all of recorded history.