Why would one be afraid of immigrants because of past treatment of original natives when chances are the immigrants are descended from people who did similar things to the original natives of their homeland? As an easy example, I'm fairly certain the original natives of the southern portions of the North American continent weren't speaking Spanish when the conquistadors showed up.
I would disagree that is is for the better of mankind. In some cases, it could be, sure. In most cases, it eventually is not. How is it good for anyone to move from one stagnant area into a prosperous one to the point of dragging it down to be stagnant as well? Never mind the affect this has the people who are currently inhabiting the area.
Plus, it would seem to be that once a level of immigration has happened in an area the newcomers no longer feel the need to respect or to assimilate into the local culture, so they maintain their own. Hopefully they are carrying over their bad habits that led to their homeland sucking enough to force them to move, because they'll eventually want to move again. In the long term, that eventually makes things worse for those that lived there before, much like you pointed out in your first statement.
We have a long road to travel before mankind as a whole can, in masses, move about freely throughout the world without there being local consequences.