Excellent point, and nobody has ever brought it up to me despite me talking this way many times! I think the "competition" angle is really good at pointing out why a land value tax is very different from a property tax here.

A property tax affects all square footage of apartments quite equally, which is very well correlated to rental prices, so a property tax is roughly equally a rise in all landlord's costs. So in the very few housing markets where rental prices have fallen to near the cost level, and there's very near zero profit in renting, property taxes could easily cause a rise in rents.

However when switching to a land value tax from the current property tax system, the incidence is quite different than a broad property tax increase. Some landlord's taxes fall quite a bit, some landlord's tax bills might increase.