And, you are part of a social contract between these other people you live with (e.g. your countrymen) where you can't just override them.
If you had your own country where nobody else lived, where you had no social contract with anyone, it would be fine for you to let anyone in (of course, you wouldn't have a country for long).
If you respond by saying that the social contract of the country (which bans discrimination based on race) supercedes the social contract of my neighborhood then I'll say that the social contract of the world supercedes even that.
You're imagining a global social contract because it makes you feel good. Pure wishful thinking.
In reality, nobody in India or Africa believes they have any responsibility to you in the slightest. They would gladly loot everything you have without the slightest concern for your rights. So it is with perhaps 85% of the world population (e.g. the part outside the West). If they feel themselves to have no responsibility for you whatever, there is no contract, because contracts have two sides.