Milton Friedman was largely in favour of freedom of movement, but he famously noted that you can have either a welfare state or open borders - but not both. (And note also that in a broad interpretation, things paid out of tax money and given away for free, like public highways, public schooling, etc., can also count as part of a "welfare state".)
The main difference between goods/services and people is that goods don't have rights, but people do. So if people are obligated to pay for the 'positive rights' of people - the "right to food", "right to shelter", and so on, then migration imposes genuine costs. (Of course, this is different from 'negative rights' - the "right to free speech", the "right to freedom of association", and so on.)