Who cares? If a sex and a race are disproportionately negatively affected, it doesn't matter whether this is definitionally prejudiced or not.
I think it does matter if you want to address the root cause of the problem.
If the problem in this case is a lack of warehouse worker bargaining power vs profit motive, then solutions to a prejudice problem will not be helpful to the workers.