Classes in society necessitate a qualitative differentiator. If you want to argue that it's fair that some people are rich while some people are poor, there has to be some natural inherent difference between the rich and the poor. Canonically in the western world it's usually "gumption" or "intelligence". You'll rarely find that argument made explicitly, but it's implicit in all discussions that presuppose economical classes.

Once that observation is made, it becomes clear that the argument is actually: This class of people is unfit for better work, and without our poverty wages they would die. In my opinion that's very similar to the idea that the "negro" was inferior to the white man and therefore it was by his grace that the "negro" was allowed to exist.