I don't think it's a mental vs. physical thing at all.
I've had jobs where I had certainty about what I was doing every day, I witnessed the fruits of my labor and I had certainty about where I stood in the organization. I never suffered burnout from them.
I've also had jobs where I was always uncertain about what I ought to be doing, where the fruits of my labor often didn't seem to amount to much (e.g. perpetual project cancellations/always changing requirements/sudden architectural direction changes) and I was really uncertain about where I stood the whole time.
I tried dealing with it with sustained periods of heavy physical activity (e.g. jogging between tasks), meditation, etc. It only helped a little.
This perpetual miasma of uncertainty and consequent stress built up over time and when it became too much - that's when I burned out.
I don't doubt that working on a farm doing heavy physical labor under similar conditions would burn me out too. My parents used to sometimes give me vague gardening tasks when I was a teenager and I absolutely hated that.