So don't worry, I'm not the guy buying the big mansion :) I'm probably your neighbor shoveling snow.
We got a taste of it when Harper campaigned on anti-terrorist paranoia, but most people just thought he was a clown. When large swaths of Canadians are out of jobs, the anti-immigrant sentiment will come to the fore. Here in Alberta it's already pretty bad - unemployment is at levels we haven't seen in decades, yet a steady stream of TFWs are still somehow getting hired, and there's no reprieve in sight.
As far as the "making homes more unaffordable" point, I could see the case for absentee foreign buyers who leave the properties empty. It's still not the real problem, but I can understand the frustration, and support initiatives like a tax on vacant property. That's a big difference from someone immigrating to the country and buying a home though! And anyway, what really makes homes unaffordable is lack of sufficient density. Sure, there's a peak where there is simply no more space, but cities like Vancouver are nowhere near it.
There is a bigdifference between an absent foreign buyer who simply takes a property out of circulation, and someone who actually moves to the country and lives in the house they buy. The latter is no longer 'foreign'; they're a new Canadian resident and IMO should be welcomed as such. Even from a purely selfish perspective (which, again IMO, we could use less of,) the country benefits significantly from skilled immigrants. Moreover, they're not what causes high house prices, except in the most marginal sense. The real problem is on the supply side. Foreign buyers are a part of it, in cities like Vancouver, but prices could still be brought down by increasing density. The general problem with that is, there's a disincentive to do that for the people who already own property, which tends to be most people in positions of power, and those to whom they are beholden. Immigrants are an easy scapegoat, but if we actually want to improve things, we need to look at policies that will do that, and I don't believe putting up a wall (literal or figurative) is one of them.