If you think this wasn't a right have y'all never wondered why states don't simply ban people they don't want?
As for the justification for why it's a basic human right
https://www.un.org/en/about-us/universal-declaration-of-huma...
> Everyone has the right to freedom of movement and residence within the borders of each state.
https://www.usconstitution.net/constnot.html#travel
> It is interesting to note that the Articles of Confederation had an explicit right to travel; it is now thought that the right is so fundamental that the Framers may have thought it unnecessary to include it in the Constitution or the Bill of Rights.