There is no "back down" on immigration, because immigration is already pretty heavily restricted. There's no way for bystanders to check someone's immigration status, so when you see someone complaining about "too many immigrants", some of the people they're complaining about are actually nationals who've been around for years. And there's no way of getting rid of them, short of violence.

We've already seen this kind of thing (http://www.bbc.co.uk/news/uk-england-essex-37227313) happening more after Brexit. Listening to people who are dangerously wrong on the facts just encourages other people to agree with them.

Whatever the right word for people who turn on immigrants like that is, it's definitely not scared. They're the ones doing the scaring.

Likewise with refugees: there's not really any policy change we can make to stop refugees coming, other than trying to fix the conflict they're fleeing. They're already fleeing death, you're not going to scare them with paperwork. Are you going to deport them (including women and children) to a warzone?

(It's rather like crime: the amount of crime you and your family have ever experienced is a number that can only go up, whereas the probability of becoming a victim has been generally declining over time. So people think crime is rising when it isn't.)