The reverse could also be said - if you don't want to deal with people deciding together that they want their community to have certain guidelines, move to the country. You can do pretty much whatever you want in rural areas without significantly impacting your neighbors.
People living together in a small area do get to decide together what they want that area to be like within reason. Creating some guidelines to create a desirable, pleasant and safe environment for them and their families is totally reasonable and has been the historical norm for most of the history of civilization.
Of course things can get out of hand and over-reaching. If the people decide together to change it though, they can. I don't think the solution is just nuking all zoning regulations from orbit.