Other white people in my neighborhood don't want to accept a black person into the neighborhood. And, I am part of a social contract between these other people I live with (e.g. by fellow neighbors) where I can't just override them.

If you respond by saying that the social contract of the country (which bans discrimination based on race) supercedes the social contract of my neighborhood then I'll say that the social contract of the world supercedes even that.