There is racism in some quarters, but there are also plenty of people who are just opposed to the free movement of people on selfish grounds.