Secondly, I gladly accept your reductio of a "general factor of fitness". If you restrict yourself to the range of world-class athletes, I grant that performance in archery will correlate negatively with performance in sprinting. This is just Berkson's paradox, or the "restriction of range" phenomenon. If you consider the entire population, however, I would be very surprised if a "general factor of fitness" didn't fall right out of a battery of athletic tests. In general, people who are faster than average will also be stronger than average and probably more dextrous too, and it makes sense to call that factor "physical fitness".
Intelligence simply doesn't work the same way, learning to draw well won't make you worse at computer programming for example.
1. Pick an arbitrary athletic contest: 100m dash, bench press, obstacle course, marathon, whatever.
2. You pick a member of the population at random.
3. I pick a member of the German national football team at random.
4. If Joe Schmoe wins, I pay you a dollar. If Dieter Schmieter wins, you pay me a dollar.
5. Repeat until convinced.
Who do you think will win more money in the long run? Will the Germans tend to win because of their high general fitness, or will it be a toss-up because the Germans merely have "high soccer fitness" which does not help (or perhaps even disadvantages) them at non-soccer tasks?
That is what I mean when I speak of a "general factor of fitness". Over the whole range of population, people who do better at one athletic task will tend to do better at any other. If you take a sample from the population, measure their performances on whatever athletic tasks you choose, and drop the results into PCA, you will find one factor which is positively correlated with every test and explains, I'm guessing, at least 3/4 of the variance. What happens at the extreme tails is already acknowledged, and does not bear on this general point.
I tried to find data for non-elite-athletes taking something like the NFL combine and couldn't find anything, but there, at least, is a testable prediction for you.
My only point about a "general factor of fitness" was that I am unfazed by the comparison to IQ, because physical fitness is a perfectly reasonable concept that captures something real in the world. So too IQ. If you think the analogy is inapt, remember who brought it up :)
Amongst the general untrained population you will see very large variance between performance on different fitness tests. Lots of broadly built overweight people who are very strong despite never having done any weight training but can't walk up a flight of stairs without breaking a sweat and skinny people who can run well without much training but barely lift anything. So I would doubt that there would be a strong general factor for physical tasks.