There are a few people who are lucky enough whose work is their hobby, and who feel happy doing it, but mostly, work is draining and stressful. And I don't mean physically draining.
I'm convinced the 'Fall from Eden' story was inspired by the invention of agriculture. Suddenly, people had to work a lot more (hunter-gatherers typically work 4-5 hours a day). Only the aristocracy inherited the privilege of having a lot of leisure time and being able to hunt(barring the US, hunting is extremely restricted).
Nutrition got worse, it took millenia of improvement till skeletons of farmers started to look as well-fed as those of hunter gatherers.
And of course proximity to farm animals and greater population introduced all those nice pathogens: tuberculosis, smallpox, flu...