However all those migrants were basically a free source of human skill - i.e. another country poured their resources into nurturing and educating them, now the country they immigrate to gets to reap all the rewards. So it's hard to see how you couldn't pay an economic price for keeping out migrants.
I also wonder how you'd feel if the economic/political situation in your own country deteriorated so much that you felt you had no choice but to migrate elsewhere - but were no longer able to because anti-migration policies had become the norm everywhere.