The two downsides I'm aware of are difficulty in transitioning to an LVT and difficulty in valuing the land.
Transitioning to an LVT means that landowners no longer capture land rents for themselves, which is a massive overnight loss in the value they hold. The solutions there typically tax only the difference in land value versus a baseline assessment. So if a lot is worth $100 before LVT and $105 after, the tax is calculated only on the $5 difference.
Valuing the land is tricky because the whole point of LVT is to tax only the location itself. So the value of any structures should be excluded from taxation, and even improvements in soil quality (e.g. on a farm) should be excluded. This is problematic because the market for bare land is significant less liquid than improved land, especially in suburbs and cities. So there isn't always good data on comparable land, and there isn't a way to hold a straightforward auction to value a given lot. Of course, most present systems of property taxation are subject to the exact same issue.