The fear isn't due to space, but jobs and the perceived toll on social welfare and other facilities.