At that point couldn’t the local government use eminent domain and just seize the property? Then build public housing units that benefit all instead of the landlord sitting on a vacant lot.
That assumes the government will make optimal use of the land. With a land value tax, there is still total freedom to do whatever you want to improve the land, you just make more money if you use it more efficiently. A landlord can still sit on a vacant lot if they so wish, or do any other inefficient thing with the land, but they are constantly feeding money into the public coffers for the privilege of doing so. Perhaps the money could be spent on affordable housing projects, but it could also be spent on any number of different things that may be more useful. The system is fundamentally flexible to a community's needs.