I'm not sure people should be forced to live with those they feel xenophobia towards, even if one grants that the feelings of xenophobia have no basis in reality.
Also you do realize that your preferred migration policy will lead to microaggressions (and possibly macroaggressions) against immigrants? If people don't have democratic means to keep out disliked foreigners, they may be tempted to try alternatives. The ballot box is ultimately an alternative to violence.
And no, we don't negotiate with terrorists. If people try violence, they go to prison. We don't want antisocial people in our society.
This is how you fuel the far-right.
I guess you must be pretty happy about illegal Israeli settlements in the West Bank? Immigrants causing existing residents to leave, right? Woohoo!