Have you ever been involved in the process of growing food? A workday on the farm is substantially longer than any office job, the only way to talk to people is to scream because they're at least thirty meters away from you most of the time. At the end of it you're drained and people who have been doing it all their lives get to live the last years of their lives with terrible back pains and having to rely on the young folks for, um, growing food, because it turns out doing manual labour for twelve hours a day isn't so easy when you're eighty. There's anything but quality time in that.
Edit: It also turns out that, with foodstuffs having to be so cheap, this kind of work was also very badly paid, which meant that except for people who owned huge amounts of land (and generally didn't work all, or more commonly any of it), people who had to do this were, if not dirt poor, in any case poor enough not to be able to afford too much. It's the fact that we lost all this chance to have "quality time" that allows us to heartily debate such matters through silicon stuff that shoves charge carriers through really thin glass tubes.
If we all had our little gardens the food landscape of America would change wildly. Japan's got tiny little gardens tucked into every corner. It's a different model of production.
This is in the context where, as ZenPro stated above:
> industrial countries spend between 9-11% of their annual income on foodstuffs and less than 5% of their time gathering said foodstuffs - the lowest in time/money expenditure for food in the history of the world.[1]
Things would change radically if you had to feed off it entirely most days. Having a small garden is great, both as a hobby and through the healthy food you get from it, but glorifying the times when you spent "quality time" working your ass off in the field is a bit far fetched...
Preparing food was a major source of such "quality time" as peeling, flaying, cooking and serving food for a large family. Quality time, indeed.