Honestly, you're right. That's why low-wage employees working in non-goods occupations (like clerks and hospitality) don't get mistreated. It has to do with access to goods, which are easily quantifiable.
I think it’s a combination of class (ie white collar workers are more similar to the management and owners), and low wage workers being considered more replaceable, thus managers feel freer to engage their baser impulses. Throw in a bit personal bias and stereotypes and voilà! You have a horrible situation.
[0] https://www.nbcnews.com/tech/tech-news/big-tech-call-center-...