I think that this is more of a philosophical question. What is wealth if it is not these things?
I think the concern is that if (some amount of) people are not effectively forced into shitty jobs that wealth will decline a lot for everyone.
Crappy jobs should be made less crappy, or pay enough that a reasonable person with options would choose to do it. All the comfort we get from people who have to do bad jobs or starve, feel unfairly gained.