> The cost of occupying this new ecological niche is that we started working all day, every day.
That's not quite true. Farming is heavy, all day long work during sowing and harvest seasons, but other than these, which take a few weeks out of a year, it's not all that much work really. Keeping animals on the other hand requires some work every day (taking them in and out of the pasture, milking, cleaning sheds etc), but it's not all day long, it's really on the order of 1-2 hours a day.
Keep in mind also that for most of the history, farmers didn't really have all that much land to farm, and each farmer would work on much smaller plot than he and his family was physically able to. The reason for this is really Malthusian. If there's excess food, you have population growth, which makes you hit the carrying capacity only in a handful of generations. Look at American colonies for example. After they were founded, the population grew mostly through natural growth, further immigration being relatively unimportant -- foreign born population rarely exceeded 10%. Yet the population increased 20-fold in only 100 years.
This means that throughout most of the history, the farmer parents only had barely enough land to sustain themselves and 2 children -- only 2 children of each parent reproduced on average, otherwise you'd have population growth, which meant war or famine (or both). The farmer didn't have a lot of farming to do, because he just didn't have enough land to plow and sow.
If you are European, talk to some old people who grew up farming. My mother grew up on a farm in Eastern Europe, and her village only got electricity in early 1980s. When I talk to her, or my grandparents, or their neighbors, and I ask them if the life was poor back then, and what was the thing they needed the most, everyone always says that they had too little land. In America it was of course much different, for obvious reasons.