Correct, which you should be rewarded for as a developer. I recommend 0% tax on improvements and 100% tax on land value.
The result is that real estate basically sells for roughly what it would cost to rebuild whatever is built on the land.
There is still a premium charged (in the form of LVT) to allocate access to desirable locations, but the revenue goes to the government/community instead of the previous owner. And the eliminating the incentive for land speculation should actually reduce overall costs.