So the solution you propose means higher consumer prices (which hurts the lower class the most) and to prevent poor people from other countries the opportunity for a higher quality of life
Worse, this policy effectively conceals the problem rather than addressing it. If fruit was too expensive for most people to buy, that would be a problem our society would need to solve. If we can pretend things are working fine by exploiting the poor, we don't have to face or solve that problem for a while.
I also don't believe your summary. It's an easy excuse to pretend that allowing the poor to come slave away in demanding conditions to provide you with low cost goods and services is actually for their benefit. Their original countries are deprived of their economic energy and the poor in the richer countries are forced to compete with a vast influx of workers.
The corollary of higher fruit prices is that the fruit workers get more money, more breaks, better benefits, and improved conditions. That's a worthwhile trade.