Maybe, many of those natives leaving every year do so because of the extra competition with the immigrants: bringing people over causes lower salaries and higher costs of housing.

What's worse is that, those leaving presumably are capable of making it in "better" economies. So in pure GDP-per capita-producing terms, you just exchanged some natives for worse performers.

That's what happens when you're not in the top of the totem of the immigration market.