What kind of days were those bad days?
A bad day at the physical work was just cold, windy, hard work, muscle aches. Luckily I never got injured and I didn't know anyone that died there otherwise I suppose my opinion of the two works might be different. But I've worked other hard physical labor; for example - ditch digging - cold, muddy, and a real workout, less strenuous - washing dishes in a restaurant, bussing tables.
Again - all preferable to programming on a bad day of programming, maybe it's because when programming goes bad I keep thinking about it afterwards, the other work I forget it right away - hell, I don't even really think about it when I'm doing it.