This is a very interesting question, and one that's being asked all over the world at the moment.
To use the example of Brexit - I sometimes feel like the 'Remain' side, in accusing 'Leave' of xenophobia & racism, painted themselves into a corner where any discussion of reducing migration was taboo.
I voted 'Remain' and am generally fine with current levels of immigration, but would I continue to be ok if these levels doubled, tripled, quadrupled? To be honest, I'm not sure.
In that sense, perhaps I'm not really much different from a 'Leave' voter, we just have different levels of sensitivity to opening our culture to foreigners.
(Or course, the referendum was about much more than immigration, but there's no doubting that it was a huge factor for many people.)