The standard counterargument is that not allowing mobility is what enabled us to have any nice places to begin with, in an "ant and grasshopper" way. Building a "nice" and affluent country is generally a multigenerational project involving plenty of individual sacrifice for the sake of a future that the sacrificing individuals will rarely themselves experience. If Norwegians knew that they all could move to any country they choose at their pleasure at all times, why would they save their oil money in a sovereign wealth fund instead of quitting work, blowing it all on blackjack and hookers right now, then taking out the greatest national credit line their country can secure to buy some more blackjack and hookers, and relocate a few 100km into Sweden once it's all used up? Knowing that the Norwegians would be thinking along those lines, what would stop the Swedes from racing to beat them onto the blackjack-and-hookers train?
Now, in reality, Norwegians and Swedes are sufficiently similar and familiar with each other that they probably would be able to predict the other's actions and jointly avoid racing to the bottom, but could you say this about, say, Germans and Egyptians, for more complex downward spirals than "burn through our assets faster than the others get to"? Add to it that the short-sighted policies that actually make the difference between successful and failed countries rarely are so straightforwardly bad as buying blackjack and hookers: more commonly, we'll be talking about economic experiments, corruption and kickbacks for interest groups, and really a handful of policies where we still can't confidently say if they were actually good for a given polity in the long run (space programmes, social programmes , infrastructure megaprojects...), but have to rely on history to naturally select for political processes that crystallise better instincts for their selection.
In less abstract terms, I think that especially many right-leaning Americans feel that Latin-American socialism gets close to being an instance of "blackjack and hookers" in the sense of the above metaphor, and in the same vein Germans feel that Levantine... sectarianism?... does. (As in, sitting down and negotiating a sustainable peace with people from other ethnic groups who might be heretics and devil-worshippers is the hardship-and-sacrifice option, whereas using any momentary advantage to enrich your friends and blow up infidels is short-term fun.)