I think if someone is in a huge house they're barely using and possibly not even maintaining, there isn't a massive issue with gently incentivising them to move somewhere smaller.
There especially isn't an issue with a redistributive land tax rather than a property tax. That would take money from corporate landlords and move it back into the rest of the economy.
You have to balance your caution with the reality at the other end - which is people living in slum conditions and/or sheds, or being homeless.