I think that's a bit of a myth. The Greeks and Romans had weightlifting and boxing gyms, but no forklifts. Many of the most renowned Romans in the original form of the Olympics and in Boxing were Roman Senators with the wealth and free time to lift weights and box and wrestle. One of the things that we know about the famous philosopher Plato was that Plato was essentially a nickname from wrestling (meaning "Broad") as a first career (somewhat like Dwayne "The Rock" Johnson, which adds a fun twist to reading Socratic Dialogs or thinking about relationships as "platonic").
Arguably the "meritocratic ideal" of the Gladiator arena was that even "blue collar" Romans could compete and maybe survive. But even the stories that survive of that, few did.
There may be a lesson in that myth, too, that the people that succeed in some sports often aren't the people doing physical labor because they must do physical labor (for a job), they are the ones intentionally practicing it in the ways to do well in sports.