As soon as we accept reality for what it is, we can begin adapting our societies and policies to maximize opportunities for individuals with the knowledge that our talents and limitations vary quite drastically.
People have a natural attraction to those similar to them. In history, the over-emphasis of particular and possibly unimportant (who knows?) traits has often been the cause for wars, genocides and all sorts of other evils.
Further, I don't think that relative Human intelligence has all that much magnitude. That is - a 'really intelligent' Human is not all that different from a 'non-intelligent' one, taken in the context of other species. And, those non-intelligent ones may have particularly interesting genetics that make them important to the species, in other ways.
Humans have done pretty well as a species with lots of diversity. Why ruin a good thing? And, do you really want Humanity to be comprised of near clones?
All to often, people start using it as justification for income disparity, wealth disparity, minority crime rates and sometimes differences in economic development between different countries. Racism and classism aren't too far behind.
I'm reminded that it is no more noble to fool a person out of his money by outwitting him, than it is to bully him out of his money with superior strength. Being smarter isn't necessarily a moral justification for greater rewards.
Similarly, we rely on the economic system using prices as an information signalling mechanism; more valuable labour is priced up, and thus supply should follow. But if the valued labour is intelligent labour, and its supply is not easily increased, then there is less of a moral case for paying its suppliers more, for that attribute alone.
I think the difference between these 2 opinions is their definition of intelligence.
I don't think I've heard anyone serious say that there was no heredity of intelligence. Could you be more specific about who says what? Perhaps you are writing quickly, but these are imaginary statements from imaginary people.
Because athletic ability is more recreational - you can say different people are good at sports, music, art, etc... there are nice alternatives. But it's not PC to imply anyone is inherently stupid. I phrase it that way because saying "inherently smarter" is OK on the surface, but the corollary is that someone else is inherently less smart. Being "PC" is all about not offending anyone.
Outside of professional athletes, being good at soccer is not much more important than having blue eyes.
Intelligence, on the other hand, is an underpinning of modern life.
The only reason we acknowledge the heritability of wealth, another underpinning, is because it is by far too obvious to deny.
(Quite the reversal from the origin of the species!)
Secondly, I gladly accept your reductio of a "general factor of fitness". If you restrict yourself to the range of world-class athletes, I grant that performance in archery will correlate negatively with performance in sprinting. This is just Berkson's paradox, or the "restriction of range" phenomenon. If you consider the entire population, however, I would be very surprised if a "general factor of fitness" didn't fall right out of a battery of athletic tests. In general, people who are faster than average will also be stronger than average and probably more dextrous too, and it makes sense to call that factor "physical fitness".
Intelligence simply doesn't work the same way, learning to draw well won't make you worse at computer programming for example.
1. Pick an arbitrary athletic contest: 100m dash, bench press, obstacle course, marathon, whatever.
2. You pick a member of the population at random.
3. I pick a member of the German national football team at random.
4. If Joe Schmoe wins, I pay you a dollar. If Dieter Schmieter wins, you pay me a dollar.
5. Repeat until convinced.
Who do you think will win more money in the long run? Will the Germans tend to win because of their high general fitness, or will it be a toss-up because the Germans merely have "high soccer fitness" which does not help (or perhaps even disadvantages) them at non-soccer tasks?
That is what I mean when I speak of a "general factor of fitness". Over the whole range of population, people who do better at one athletic task will tend to do better at any other. If you take a sample from the population, measure their performances on whatever athletic tasks you choose, and drop the results into PCA, you will find one factor which is positively correlated with every test and explains, I'm guessing, at least 3/4 of the variance. What happens at the extreme tails is already acknowledged, and does not bear on this general point.
I tried to find data for non-elite-athletes taking something like the NFL combine and couldn't find anything, but there, at least, is a testable prediction for you.
My only point about a "general factor of fitness" was that I am unfazed by the comparison to IQ, because physical fitness is a perfectly reasonable concept that captures something real in the world. So too IQ. If you think the analogy is inapt, remember who brought it up :)
Amongst the general untrained population you will see very large variance between performance on different fitness tests. Lots of broadly built overweight people who are very strong despite never having done any weight training but can't walk up a flight of stairs without breaking a sweat and skinny people who can run well without much training but barely lift anything. So I would doubt that there would be a strong general factor for physical tasks.
Intelligence, I think, can broadly be defined as "the ability to score well on intelligence tests." The most common is supposedly correlated with freshman college survival. History is full of people who were considered "dull" as students but later shook the world in a way that means they had significant intellect.
It wouldn't be a problem except we sort people into piles and this is yet another sorting mechanism.