"Anti-immigration voters aren't driven by actual immigration."
"and vote accordingly in places where there are few immigrants, in comparison to places where there are lot of immigrants."
No. The number of migrants to the UK is well publicized, and it's quite high, moreover, EU migrants have considerable privileges within other EU nations (i.e. benefits, voting), which would be truly odd for most citizens of the world. (Imagine Canadians voting in US elections).
The population is well aware.
As far as 'those in areas with fewer migrants' voting more for reduction as evidence of 'lack of knowledge' is not quite right either.
People in non-urban areas simply have very different attitudes towards migration - and this is avoiding the fact that many people in Western urban areas are themselves immigrants - or children of immigrants.
Toronto, for example is 49% immigrants, and 75% first generation Canadian. Imagine how they would feel differently than say, Hamilton, Ontario, which has considerably fewer immigrants?
Clearly, there will be differing attitudes.
I'd urge us not to belittle those with whom we may disagree, or else the pendulum will just swing further: were the EU to have effectuated more sensible immigration reform over the years, consistent with the will of the vast majority of EU citizens, then there'd be a lot less turmoil, no Brexit, and there wouldn't be this massive rise of nationalist parties.