> And there's no way of getting rid of them, short of violence.
And that's where people become really-really cautious about immigration. You can't really change your mind later - after a few years your undesirable neo-compatriot can't be got rid of. It's irreversible. People fear irreversible decisions and don't trust politicians on those.
So the only shot you have is not letting people in at all. Even if it means turning down honest good people.
If you want to make people more immigration friendly, figure out how to track and deport existing immigrants when things go wrong. Not just tell other people what to think.