US and EU citizens could visit most places on a whim, but leaving and staying permanently elsewhere could be a lot more difficult -- even if you were an American who didn't want to go farther than Canada. (After Trump's election, there were more than a few Americans who found that a very unpleasant surprise, for some reason.)
And as we're seeing on the US southern border right now, the rules are very, very different for people who want to pick and leave, say, Guatemala.