Or, to put it another way: what gives you the right to exist in a public place in the country you live in, but not a poorer person?
However, an argument can be made for the self-determination of groups as well, particularly their freedom of association (or lack thereof).
Then there are more pragmatic matters. You can't simply let people starve if they can't provide for themselves. Providing for everyone who chooses to stay at a place but can't actually afford to do so might just not be economically feasible.
Sure, over time these differentials will be evened out because people won't come to a place if it doesn't make economic sense to do so but simply opening all borders without taking possible consequences into account could lead to quite harsh outcomes.
If you don't apply the social safety net to the incoming immigrants, I don't see how you hold any higher moral ground than people who want controlled migration.