If we can’t have a discussion without people accusing others of being racist or xenophobic instead of refuting their arguments with data and concrete analysis, then we can’t have a discussion at all.
And yes, the majority (and growing) in this case would be called racist and xenophobic, because they have not been shown a single fact that runs counter to their argument. Not even an attempt. And you know what? They’re probably right.