> we're doing something valuable to society because someone in society is willing to pay for it

This is where you are completely wrong. I'm not even going to bother explaining, because there are too many obvious counter-examples.

The ire is because they are saying this in a context where they are implicitly talking not just about their personal preferences, but furthermore about the opportunities that should or should not be made available to others. They are saying this in a context where it functions as an argument against Basic Income, on the grounds that leisure is bad and everyone should be made to work as much as possible, to provide as much for society as possible.