From an economics point of view, we should allow the free movement of labor (people) because we already allow the free movement of capital (money) across borders. If my understanding is correct, this is one of the pillars of the European Union.

However, I think looking at it as a basic human right is more appropriate. Restricting people crossing borders is simply institutional racism. Countries are granting permission to enter/not enter based on the person's race. If this were any other context, people would be screaming racism. But because it's become so typical over the past 150 years, nobody calls it for what it is anymore.