Either can be taken to an unhealthy extreme, but on balance the human body benefits more from physical exertion than the lack of it.
Sitting in an office desk for 30 years can't be good for you either, but if it's in a position to keep my brain active (even if my muscles aren't), I'd take that job a thousand times before going back to pure labor.
In fact, one of the most memorable parts of those summer jobs was creating a shoulder rest for the 25 lbs bottles to sit on, out of discarded plastic. I honestly can't remember whether it worked at all or not, but I definitely remember being far more interested designing and cutting this piece of plastic than carrying bottles of water.
Do you find programming jobs intellectually stimulating beyond junior level? Maybe I have really bad luck, but all that I seem to be able to find - and see my friends doing - are the computer equivalents of lowest-level construction work. I usually have to step into other people's competences (e.g. contributing UX or even domain-level ideas) to get anything interesting from them.
So are sedentary jobs. Obesity, heart disease and everything else that goes with them are no joke.