I run into lots of climbers who took to climbing without ever being much interested in any other physical pursuit before hand.

The other thing is that there are certain adaptations one has to make as a beginner to progress that you don't really get anywhere else. It somewhat levels the playing field between athletes who are generally good at sports, and those who aren't super physical.