Since that would reduce demand in the area, there's a real risk that we'd infer that the neighbourhood is more inherently undesirable and drop the rates there, even though we shouldn't.
I think you're really basically just saying "but no one would do this". People would absolutely do this. People even do this today in inter-group conflict about who is to live in specific neighbourhoods.
Imagine a group that chucks rocks at everyone else who lives in the place where they want to live, then try to buy the people living there out. It's the same strategy.
Also the asshole still saves on the purchase. The taxes go back up, but the sale price of the land was artificially low still. Which benefits the asshole and hurts the original owners.
The only way it doesn't benefit the asshole is if the property is worth zero dollars to sell.
The LVT assessment doesn't have to consider whether it is a "desirable place to live" or what the neighbors are like. Only the potential for the land.
A key factor in LVT is the local control of the process. That is the part that can be more easily swung by coalitions with money/power.