Every time I've heard this sentiment or something related expressed by a friend, it's been from someone whose use of their leisure activities are utterly boring. When they open up a new chunk of regular free time, they don't pick up a new instrument or sport or online course, they just do their leisure activities a bit more often: go to the same bars a bit more, watch a bit (or a lot) more TV, go to brunch a bit more often. (To be clear, I enjoy all these things and don't think they're inherently boring, but think that there's obvious a point at which marginal returns are diminishing)
By contrast, I and many people I know aren't already in a surplus of leisure time. I've been debating re-orienting my career to reliably require less than 40 hours per week because I simply have a dozen other interests I want to pursue. I enjoy my job, but 40 hours dedicated to _one_ of these pursuits to the detriment of the others is a disproportionately huge amount, especially given that it's 40 daylight hours. Given the long list of time-consuming things that I already find rewarding, I can't relate to the concept that being a slave to the drudgery of menial tasks is somehow something that would be _missing_ from my life if I managed to remove it.
To reiterate, I do think that a shift in the economy that leads to increasing outsourcing of menial tasks to an underclass is an issue worth thinking about in terms of what it says about the way our economy is structured and why said underclass is in that position. I just don't buy the self-flagellating claim that doing X hours of penance a week on things you'd rather not do is somehow a moral good.