I don't classify it as a smear - it's true: control on immigration is, on varying levels, direct xenophobia. Most border policy is straight up prejudice against people based solely on the fact they aren't a citizen here, to different amounts based on the location they're coming from and the nature of their business in the receiving country.
Labelling people who voted to restrict border controls prejudiced against those who come from abroad isn't some horrific smear insult, it's merely stating the facts: they don't like "immigrant type X" and have acted accordingly.
I was also explicitly referencing my experience with those who I know who are outwardly Leave. Not once has any specific policy come up within their reasoning - broad feelings have been far and away the driver.