You might find useful this overview of the unconstitutionality of Federal immigration restrictions: https://tenthamendmentcenter.com/2017/12/13/the-constitution...
The purpose of the interstate commerce clause was to prevent states from enacting tariffs or banning goods from other states or nations. There’s plenty of analysis of the origin and intent of the commerce clause online if you’re interested.
10th amendment: The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people.
It’s ironic that one of the original grievances in the Declaration of Independence was the king restricting immigration to the colonies.
Freedom of movement is a common law right going back to the Magna Carta: It shall be lawful to any person, for the future, to go out of our kingdom, and to return, safely and securely, by land or by water, saving his allegiance to us, unless it be in time of war, for some short space, for the common good of the kingdom: excepting prisoners and outlaws, according to the laws of the land, and of the people of the nation at war against us, and Merchants who shall be treated as it is said above.