I mean Neil Armstrong literally smoked and did not "believe" in excercise so they were absolutely not the most physically fittest people. They were just freaks in terms of enduring a lot of stress tests. Physical endurance is just one aspect they train for. Other aspects were much more valued like them being military flight pilots/smart enough to understand the systems/mentally strong enough to not break down etc. You were not selecting for absolute raw fitness for the apollo missions.