Physical labor correlates to social status not intelligence.
It's a huge mistake to look down on and make assumptions about someone just because they have a "bad" job.
A friend of mine studied enology (wine making) at Cornell and his job is about 50% business management and 50% wine making. Some days he has to wear a suit to work, other days he comes home in a wine stained smock. It seems like a good balance.
I have friends who work in various construction fields, so I know it can be miserable and cold and dirty and hard on your body, and that people tend to romanticize outdoor jobs. Still, I've never been one to be bothered by cold or reasonable amounts of rain, as long as the appropriate clothing is available.
After spending a few days bolting heavy steel pipes into place in the scorching sun I realised I quite like the comfort of the office ;-)
Though it is fun to go onto the absolute front lines and get your hands dirty and the feeling of satisfaction on completing a physical job can be very high.
Working on the house has so far given me a lot of satisfaction (also some stress and shorter days). Any work we do on the house is work done for us. We can't do everything ourselves. We hired an electrician (a friend) to do some electrical work and got a company to do floor grinding in one of the rooms. Her father helped us oiling the floor, my father helped us when we redid the insulation in a part of the house (still ongoing). We planted some garlic and some raspberries. We're going to plant some strawberries soon (not really the right season, but we got them for free).
Working on your own house and garden could be an alternative to a career change if you feel the need to work with your hands.
Your parent's opinion: This view is incorrect, people with higher social status, not higher intelligence, receive jobs with less physical labor.