There's no innate human right to freedom of movement within other societies.

Humans are individuals, individuals form into groups, and those groups when large and cohesive enough form into societies, civilizations, and now nation states. If those groups are to exist as a "group" they require these shared norms, values, culture to remain a viable group. If part of that culture is to exclude outsiders on the area of land on this earth that they control, then who is one individual or many individuals to say that this is invalid?

This presents a lot of opportunity for places like the United States where immigration and freedom of movement is more accepted - so you get more immigration and more growth. But that is something citizens of the US have decided, not external actors.