I think the misunderstanding comes from me using the term "fit" without actually specifying the precise meaning. I meant it more as having better general health in the colloquial sense(e.g. not having life-threatening congenital diseases) rather than fitness in the Darwinian sense where certain characteristics allow one to prevail in natural selection.
Do you have any suggestions on what I could read to better understand the issue at hand and get a better idea of what I'm trying to talk about? Perhaps a good book on the history of ancient humans?