The following men won gold in a solo swimming even in 2008.

  http://en.wikipedia.org/wiki/Michael_Phelps
  6 ft 4 in
  http://en.wikipedia.org/wiki/Alain_Bernard
  6 ft 5 in
  http://en.wikipedia.org/wiki/C%C3%A9sar_Cielo_Filho
  6 ft 5 in
  http://en.wikipedia.org/wiki/Park_Tae_Hwan
  6 ft in
  http://en.wikipedia.org/wiki/Oussama_Mellouli
  6 ft 3 in http://sports.yahoo.com/olympics/beijing/TUN/Oussama+Mellouli/217537
  http://en.wikipedia.org/wiki/Aaron_Peirsol
  6 ft 4 in 
  http://en.wikipedia.org/wiki/Ryan_Lochte
  6 ft 2 in   
  http://en.wikipedia.org/wiki/Kosuke_Kitajima
  5 ft 10 in 
  http://en.wikipedia.org/wiki/Maarten_van_der_Weijden (10 km marathon)
  6 ft 7 1⁄2 in 

Height is not the most important factor in swimming, but out of 10 people, only one was under 6' and he is still taller than average. Now look at something like size of hands and they are going to stand out even more. Practice may be able to get an average person to the 90th percentile in most things but once you start talking about 99.99% DNA becomes extremely important.