Even now, there's no shortage of travel blogs talking about moving to a new country and the vast majority fail to address this key detail (usually it seems they're written by a spouse of someone from said country).
It would appear it did indeed used to be easier to travel pre-WWI, at least in terms of paperwork. There one was an understanding we should abolish passports: http://uk.businessinsider.com/when-governments-wanted-to-get...
Unfortunately, I strongly suspect that the main reason we keep immigration controls around is that not long after we brought them in to force, travel got really cheap. In 1880 traveling internationally was a hell of a lot harder, and one supposes there weren't as many people doing it (even proportionally). There also was just an awful lot more Earth to go around - a world with 1 billion people is a lot less crowded.
Even now, most countries are fine with rich people entering. We just stopped being able to use "they came 5000 miles to get here" as a filter.
It's kind of interesting that even though I'm fairly left, this is one of the arguments against a strong social safety net that I can buy. When you view every potential immigrant as a potential expense it gets harder (even if only politically) to let people in without trying to filter out those likely not to be net contributors.
I am also surprised that any nationality (except Americans) would have an easier time getting a US visa than an EU one!
https://www.schengenvisainfo.com/who-needs-schengen-visa/
Among the people needing visa are citizens of India, China and Russia. I don't know which world you count them in, but that's helluva lots of people.
"Svalbard has no border controls, and citizens of the 41 signatories of the Svalbard treaty (including such unlikely countries as Afghanistan and the Dominican Republic) need no visas or other permits to visit – or even work – in Svalbard."