For instance it gives property owners in a residential neighborhood an assurance that a neighboring building can't decide to convert their rooftop to a nightclub potentially disturbing the neighboring buildings with noise, foot-traffic, car traffic, drunks, trash, fights, etc. If you gave individual landlords the right to make these decisions without wider oversight, you'd run into a lot more issues like these. Sure - today they can petition to override existing zoning regulations and that sometimes happens, but residential issues are larger than just an individual apartment or building.
Maybe it's just that I get peeved when people get all "landed gentry" and start thinking that somehow because they are privileged to have been able to buy a piece of property that they have a moral right to control the lives of their neighbors.
That said, I support regulations "within reason" and I suppose it all comes down to a quantitative difference in where we draw that line.
You can make an argument about not restricting the free market, but shelter is such a basic human need that I think it merits a healthy amount of regulation.