Suppose you're of the view that a mass influx of people into your country that are willing to work for less than the locals are will do harm to the economy. If one country's economy is weaker than yours and is a source of immigration from people seeking greater opportunity (and who'll do anything to get it, including undercutting locals), is it racist to want to stop immigration from that country?
It's definitely protectionist and perhaps xenophobic but I don't think it's racist. (There is a clear distinction between xenophobia and racism - xenophobia is purely about an unjustified dislike for those from other countries while racism is an unjustified dislike for those who look different to you)