The problem is when there are confluences of conditions that systemically prevent someone (or their 'class') from having the mobility to exit their condition, even when they have enough merit to otherwise warrant it.

Take for example indentured servitude, or third world sweat shops. Sure, working for $0.28 a day in a (dangerous, unsanitary) factory is better than subsistence farming. But if this opportunity is enough to feed a family but not enough to lift the family from a condition whereby it can exit from needing to work in the factory, how is this so very different from perpetuated servitude?

Take serfs. While serfs were 'bound to the land', the condition of the serf was voluntary and mutually beneficial: Wikipedia reads "Serfs who occupied a plot of land were required to work for the Lord of the Manor who owned that land, and in return were entitled to protection, justice and the right to exploit certain fields within the manor to maintain their own subsistence."

Certainly it was a better deal for the serf to labor underneath a lord than live in the wild - especially since all fertile lands had already been captured by lords. And indeed, you see the language and argumentation during the time period repeat a variant of your argument: both the serf and the lord benefit from the situation and therefore it is good. Similarly, whites made similar arguments about black slavery throughout the history of American slave trade.

But a situation where both parties have no benefit by changing their 'local' stratety by small amounts represents a local Nash-like equilibria at the best - and not anything one could argue is a 'global' optimum. And in fact, if one truly buys that markets are meritocratic, one should be willing to subsidize the equality of opportunity so that as many people start off on an even footing as possible.

In sum this is to say that, like Braess's Paradox, locally rational decisions lead to local but not global maxima - whose gaps can be seen to be gigantic from history. When we discuss opportunities for labor, wealth and quality of life, this becomes an ethics question.

It is not so clear, at least to me, that the principle of trade (especially given its other major problems not covered here) should be prioritized over ethical considerations. Quandaries like this do lead us to ask what sorts of wages and opportunities are 'fair'.