Mine comes to the same conclusion via a different route. I don't want my government telling me where I can or cannot travel to or decide to live, and I want all other governments to do the same.

It's a romantic rhetorical cudgel in this context. And it's actually quite complicated, though easy to say.

I mean, I think that's a noble position to take; I share it, to a significant degree. The problem is when you start getting into the nuance of the modern world, where everywhere has a record of who you are, what you've done, where you've been, etcetera. The existence of that data means the masses will want to judge immigrants by said data before permitting entry, which means bad actors will leverage said data to persecute those they hate.

In principle, I am 100% with you. Pragmatically however, it's significantly more complex.