For example, you don't want people doing things that poison groundwater for the neighbors, right?
Anyway, letting property sit idle benefits the landowner but causes issues for neighbors, local government and the free market. It means businesses in those locations get less customers, and local governments less sales tax. If the property is not cared for, there can be additional issues that lower the value of other's homes.
It causes rental prices and home prices for buyers to inflate because of the market distortion, leading to boom bust real estate problems.
Boom and bust effects homesellers and homebuyers, but also, given how often property tax plays a role in government budgets, dramatically affects local and state revenue, (and federal revenue in countries with a federal property tax).