For me, the problem is more when people think: a) the different-skinned guy down the street is causing this problem; b) voting for right wing guy can stop globalisation. Brexit showed that the people most likely to vote on immigration/kick out foreigners basis were also those with the fewest immigrants in their region.
More left-leaning people seem a bit confused about inequality/poverty vs globalisation - inequality globally is on the way down, which is why it's on the way up locally (you're losing to the guy half-way around the world, some of your neighbours are not losing to their equivalents... yet).