Do you "deserve" whatever wealth you have when it could be liquidated and save one live for every $3,500 [1]?
It's easy to be generous with other people's resources. Being as virtuous as you imagine you are is expensive.
The interesting bit is that there are few who make this argument at the low end. The idea of open borders for unskilled folks is way less popular.
The net effect of this, it seems to me, is that while there's lots of support for me facing more competition, there's no support for more competition in the labor I buy.
Now, one could say that is reasonable, just because I make a lot more money than I need to live, which can't be said for less skilled workers, but it's an interesting dynamic.
"The gains from eliminating migration barriers dwarf---by an order of a magnitude or two---the gains from eliminating other types of barriers. For the elimination of trade policy barriers and capital flow barriers, the estimated gains amount to less than a few percent of world GDP. For labor mobility barriers, the estimated gains are often in the range of 50–150 percent of world GDP." [1]
A typical example:
"Take a male construction worker in the capital of Ghana. There isn’t much you could do to greatly raise his economic productivity in Ghana; access to better tools or training might make him modestly more productive. But if you let the exact same person emigrate to work at a construction site in any big U.S. city, his economic productivity would rise roughly 700% to 1,000%." [2]
[1] Clemens, Michael A. 2011. "Economics and Emigration: Trillion-Dollar Bills on the Sidewalk?" Journal of Economic Perspectives, 25(3): 83-106. https://www.aeaweb.org/articles.php?doi=10.1257/jep.25.3.83
[2] http://www.cgdev.org/blog/trillion-dollar-bills-sidewalk-why...