I've have thought about these lines before. Free movement of people is the final freedom. First they freed up the movement of goods (once upon a time you couldn't just order a product freely from another country). Then they freed up the movement of capital (likewise once upon a time you couldn't freely invest your money in a foreign country). If they freed the movement of people then you'd have a sort of anarchistic democracy. People would vote against war by leaving (most do already, but just end up in refugee camps on the borders), vote against poor economic policy by leaving, and vote for free enterprise by emigrating.

But over time I've come to see that these are the reasons this final freedom WONT be allowed any time soon. There is great power and profit in the hands of the state when you restrict the free movement of people. For instance, do you think Assad would be all for an open border around Syria at this point in time? Would that strengthen or weaken his power base? Likewise for all countries. Opening movement across borders is good for all free citizens of the world. But very bad for politicians and their cronies.

On top of this politicians in democracies have at their disposal an amazing vote winning tactic. They can at any time exploit the in-group/out-group evolutionary bias, invoke a terrible, external threat, and most of the fearful masses will fall for it and vote for them and their get-tough-on-the-immigrants platform. And so, although I share your vision of a more free world, it just isn't going to happen any time soon.