Why would you not tax the improvements? An empty lot isn't housing, if you're trying to disincentivize the hoarding of houses, wouldn't you want to target houses and not land property?
I like the idea of a different incentive structure: no taxes on the sale of a newly built housing, high property taxes on houses with an exemption for primary residence, and leave the existing property tax structure for business and fallow properties alone. This way you encourage new housing being built, encourage ownership and not renting out housing as a business or sitting on houses to flip them.
Tax the land high enough, and you'd be losing money to just sit on it hoping it appreciates in value. You'd need to build something (like housing) to make a positive return, so you're incentivized to do so.
If you tax houses higher, people are directly incentivized to not build any, and instead just hold vacant land.