If you accept that Usain Bolt is a astronomically rare genetic freak (which you just have) then you accept that some people have innate genetic factors that make them better at a sport than others.
Because people exist on a continuum and not in discrete boxes that means for a given subset of people, some people in that group have innate factors that advantage them over others in a particular tasks, just not at the extreme level of Phellps.
That means some people have 'talent', or as I like to say physiological factors that give them greater maximum potential.