I don't buy it. You only need to workout about 30min a day, 3-4 times a week to stay in decent shape. Maybe an hour if you include prep and showering. That's only a few hours of your leisure time a week you might be missing out on...
The biggest difference is the leisure time that accrues to you by retirement. On top of having more money saved, you're almost certainly in much better physical shape than someone who engaged in physically demanding activity for 8 hours a day, 40 hours a week, for 40 years.
Have you seen professional acrobats? They look 15 years younger than they are.
There's a tremendous difference between physical fitness and running your body into the ground through long days of extraordinarily difficult labor.
I suspect acrobats self select as people who look young, as that seems correlated with agility. Not withstanding that, the most common advice my high school coach gave us was to not over-practice. (I ran cross country, most of the more dedicated members had at least 1 season where they had to drop out from over-exerting themselves).
Or you could go even further and incorporate workout into your leisure time: play a sport!