You make a lot of assumptions here.

Instead let's try a hypothetical. Perhaps someone has a historically oppressed indigenous background (perhaps they even have brown skin) it is possible that they may not want any more immigration into their lands.

In my opinion that would be a very understandable position for them to take given the historical circumstances.

The economic argument isn't as clear cut as you suggest either. An economy with excess supply of underutilized labour isn't going to see many gains from immigration.. There's market distortion to consider with regard to housing supply as well, and plenty of other reasons someone may take a valid position against immigration without being racist