Go ask anyone aged 45 or over, who does physical labor - construction workers, electricians, farm workers, flatbed truck drivers. They will all tell you of the severe toll on their muscles, bones, and most specifically joints. Shoulders, elbows, hands, knees, and backs get to a point of being functionally disabled.
So, yes, physical labor, even with full OSHA support, will wear your body out. Alongside all those structural injuries seems to come a penumbra of other illnesses, like heart and kidney disease.