The thing is - normal immigration and refugee immigration are two different things.
The refugee problem cannot be solved by "not accepting refugees". That's not a viable solution.
What should be done differently? Building huge internment camps? Would the human suffering really be smaller in those camps than it has been in society as a whole now through crime etc? I doubt it.
Those who argue that the wave of refugees caused so much problems rarely present any solutions.
Non-refugee immigration on the other hand (of which Sweden had had lots) thats a different matter. But somehow that's not the focus of this recent debate.