Think of houses sold to outsiders as a form of export. It's the best form of export: the good you are selling does not leave the premises, and the local community continues to collect taxes on that. The outsider does not consume government services as much as a local, so, what's not to like? Even better, when those outsiders visit, they spend much more on local businesses, because they are tourists.
The more houses that are empty most of the year, the better for the local community. Because that's tax money that gets collected at very little cost for the local government.
The problem, of course, is the artificial barriers to build more. People have this idea that the land is fully built up, there no way to build more.
That is so far from true. I live in one of the most densely populated neighborhoods on Earth (17k people per km2). Recently a 67-story tower was built across the street from my building. The neighborhood does not feel any more or less crowded than before. If they want to build some more, they can build more. There is (almost) no lot that's not built up, but you can take down a 12-story building and erect a 50-story one.
So, here's my thesis: let builders build as many residential units as they can (provided they are safe, or course). This way the house prices will come down to earth. People will be able to move from the rural areas to the big metropolitan areas, where the jobs are. They can't move right now because the housing costs are just so damn high.
Such a migration would result in both increased GDP growth, and lower impact on the environment. Much lower. I used to live in a very dense city before (3600/km2) before I moved to my current incredibly dense neighborhood. My energy footprint went down maybe by a factor of 10. Easily. Grocery is across the street. Before it was a 15 min drive.
Can you believe that? More affordable housing, higher GDP growth, lower environmental impact? There's no closer thing to a free lunch in our modern society.