And yet there are so many more people who came from as fortunate or more fortunate means, including those who went into mathematics, but are not as talented as them.

Michael Phelps is not just a product of luck, either. You could train your entire life and not be as good a swimmer as him. You'd have to redefine luck to include genetics.