I mean, of course, we all would like to be able to travel without restrictions ourselves. The concern for people in developed nations is what would happen to their quality of life if people from poorer nations could freely migrate.

Europe has ~750 million people, and even with current policies (where migrants might drown when their boat sinks while the Greek Coast Guard looks on and laughs) millions of migrants try to enter Europe each year.

The US has ~340 million people, and even with current policies (where children might be separated from parents and placed into cruel detention centers) millions of migrants try to enter the US each year.

If movement was free, how many hundreds of millions would pour from Latin America, Africa, the Middle East and South Asia into Europe and the US? The 3+ billion who live in the tropics are only going to become more likely to try to migrate as the climate continues to warm.