The Georgist LVT is much more bizarre than that. Your taxes don't go up as a result of the improvements you make to your land, but rather they go up as a result of the improvements your neighbors make to their land, if such improvements lead to increased perceived market value for the entire area.
Ad valorem taxes already do this to some extent, but are tempered by assessment rules and annual caps on increases, to prevent people from losing their own property just because of external fads or the subjective feelings of others.
I guess I just don't get what benefit the LVT proposals are bringing to the table. Ad valorem taxation already exists, it's already well-developed enough to have safeguards against the worst-case outcomes, and all the Georgist LVT would do is reinvent the wheel, but this time using design principles anchored in a fundamentally incoherent ideology that brings along a lot of other dangerous implications.
I have to ask what real-world problem any of these proposals are a solution to, because from my vantage point, I don't really see one. I see devotees of an abstract doctrine going out of their way to contrive scenarios to apply their doctrine to.
In short, ad valorem taxes land+improvement and LVT just taxes land. Various tweaks that have been made to ad valorem systems could also be made to LVT and don’t constitute a difference between ad valorem and LVT.
The objections you raise about how your neighbor’s development raises your taxes apply to both LVT and ad valorem tax regimes.