Bear in mind the level of immigration to the UK has risen very rapidly over the past 10-15 years and it has not been evenly distributed across the country. There are places where people have seen the racial makeup of their neighbourhoods changed greatly, where once they could reasonably walk down a street and expect to hear conversations in their own language now it won't be unusual for them to not understand half or more of the conversations going on around them. A lot of people find that quite alienating especially considering it's happened so quickly.
When people talk about immigration that's really what they mean, I do genuinely believe that the UK is not a particularly racist place that the huge majority of people even those who cite immigration as an issue believe in "live and let live", but I think they feel that tolerance has been abused by a system that says free movement of people is meant to benefit them but so far as they can see it does not.