In USA, the nation is mixed, heterogenous, and thus being anti-diversity unavoidably involves some discrimination of your fellow citizens, being an asshole to some groups of them; and justifying this discrimination tends to require some racist arguments.
In homogenous countries like Japan, the issue is different - you can easily consider other races/ethnicities as equally good/valid/etc while at the same time being anti-diversity, favoring near-zero permanent immigration; i.e. a simple status quo position "the other races are nice, let's visit, chat, trade, exchange experiences but let us stay here and let them stay there" is feasible, unlike USA.
In USA, acknowledging "there's us and there's them" race separation divides the country, in Japan the same thing can unite it. Treating members of another race as guests that are different/separate from your group is reasonable in homogeous nations and horrid in "melting pot" nations.