The same argument applies to any job: in most scenarios, it pays what it's worth to society at the market clearing price. The government can interfere via licensing, minimum wages, quotas, etc; but broadly the job pays what it's worth.
Pay and worth are different, just like price and value are. Garbage collection is worth a lot, but its pay is determined by market dynamics. As the number of unemployed increases, it will pay less.
Not everyone wants to work in an office; some people really like working outside and being indoors is like prison.
But you're right, the job isn't as bad as it used to be.
I remember being in a meeting once when a manager looked out at the guys on a scaffold cleaning the windows and remarked "at least we don't have to be out there like them." My first thought was that they were probably thinking "at least we don't have to be inside like them."