Yes, there are two reasons that interact for this:
1. The population feels the country is full
2. The population feels the government is systematically biased in favour of huge quantities of immigration and doesn't listen to their concerns
Both feelings are pretty much rooted in reality. The feeling of "fullness" comes from the fact that scaling infrastructure is constrained by physical and social limits like how fast new doctors can be trained, capacity of railway lines etc but the number of people that will turn up if allowed is unlimited. So everything in the UK is constantly on the brink of falling over due to being constantly at max load.
The second feeling is even more powerful. Immigration has become totemic because for so many years the political elite told voters they couldn't control immigration, that somewhere along the line - without any announcement or debate - the entire topic had been removed from the democratic realm entirely, that the UK had to accept anyone who turned up from anywhere and changing that was not only impossible due to the EU, but racist/xenophobic/immoral.
Tell large numbers of people who are concerned about overload that their concerns are racist and they should feel ashamed of themselves, that there should be no limits anywhere despite the obvious physical impossibility of such a policy, and it's a recipe for making the electorate very obstinate indeed.
However it has nothing to do with actual xenophobia. The UK is by far the most racially diverse country in Europe.