It's quite likely not ex-locals who own it, and in any case if they no longer live locally, they're not paying local taxes, or supporting local business.

I agree, income from property should be heavily taxed, it's essentially private profit from the land value, and that value derives directly from it's location and the strength of the local economy, rather than any effort or contribution from the owner.