To try to illustrate, I'll switch the analogy to my own country, which is Canada, where we fly in Mexican labour to pick crops.
If we didn't fly in migrant labour, we'd have to pay Canadians, who are generally a spoiled, lazy, picky bunch (at least compared to the sort of Mexicans who fly here to work, who obviously are not lazy nor excessively picky). This would drive up the cost of fruit, which is perhaps fine. It would also drive up the cost of unskilled labour in other areas, perhaps construction or retail, because there would be less competition for those jobs. That's good news for the unskilled workforce in the rich country. Great.
Of course, not only do prices go up, but actually just less food will get produced. You could argue that that's good, or bad. Fine.
However, now all those Mexican labourers don't have foreign work, so they compete for work at home. That drives down the price of unskilled labour in Mexico, and if there's a minimum wage then it drives up unemployment. The industrious folks who were savvy enough to get the Canadian jobs will take the next-best jobs, displacing slightly-less-hard-working Mexicans from their jobs, and so on. Of course, this is also foreign cash that is no longer coming into the country, so it has effects on the balance of trade, i.e. on the ability of Mexico collectively to import nice things that they aren't making internally. That's probably bad news on a decade-long timescale, but it might eventually lead to a more robust manufacturing economy, I don't know.
So, in summary, suppose Canada stops using Mexican farm labour (or Britain stops using eastern European farm labour). What are the consequences:
All Canadians get more expensive food.
Unskilled Canadians get more pay, probably offsetting the more expensive food for those people.
Less food gets grown in Canada overall, probably, or at least less luxury food.
Unskilled Mexicans are fucked over hard, especially those who are most vulnerable.
Rich Mexicans can hire poor mexicans at even more exploitative rates.
In the medium to long term, Mexico might have a worse balance of trade, but then again maybe that results in positive industrialization in the long term.