LVT is generally conceptualised as proportional to the value of the land, not it's area.
You could reform zoning, but its not a requirement
If I have a SFH zoned for SFH in the heart of manhattan next to skyscrapers, the value of the land isn't based on the skyscrapers, it is based on what you can put on it and you can prove that by selling it for which nobody is going to pay as if you could put a skyscraper there.
In urban or higher density suburban areas, I think you're mostly right. But in lower density areas, a lot of the land is multi purpose or has few restrictions. So technically, you could build apartments or homes on land that is used as agriculture. However, there might not be demand for it in that area. Essentially, the tax-by-zoning works well in the situations where zoning is restrictive and demand is high. But in low density areas where the actual use is not well defined with a single zoning use, the value would be potentially arbitrary or inaccurate (making property tax a better measure of value).
It seems current actual land usage through property tax better accounts for the way the land is being used, is more stable, less arbitrary, and implicitly complies with the existing zoning.