It does seem pretty fundamental to me. I do think communities should be allowed to organize themselves in basic ways that are not discriminatory against protected classes.
Zoning is pretty important. A neighbor recently subdivided his large lot into 3, and having enough frontage on the street meant they could do it without zoning board approval. The houses they are building were designed precisely to push to the max against every possible zoning bylaw; setbacks 1 inch above the minimum, building height-above-grade 1 inch below the maximum, lot coverage ratio 1% below the maximum, etc.
Then their proposed septic failed perc testing at ground level, and since they didn't want an "unsightly mound" in their backyard, they brought in about 80,000 cu yards of dirt and raised the whole lot up by ~20 ft! Because the bylaw was written as building height calculated against finished grade, not natural grade there was nothing stopping them from creating their hill. Well, they did have to build a substantial drainage system around the perimeter because it's illegal to increase stormwater runoff from a lot in the natural vs. finished state, but we'll see how long that lasts...
Builders will cut every corner, push every limit, and develop to the absolute highest density to maximize profits. Of course building codes and zoning bylaws make housing more expensive to build. Getting a glimpse at what builders will do given the chance, to literally my back yard, I really, really wouldn't want to see what would happen if zoning boards could be overridden by outside developers wanting to "increase supply".
For example, most of the new construction in SF is rental units. You can't buy it, you can't own it. It's going to be mostly corporate housing, run by corporate property managers, with cookie-cutter units, probably poorly run and poorly maintained.
I've been on a condo board for ~10 years. Building a community is hard work! But as the rental ratio in the complex climbed from 0% to 32% over the last 10 years, there's a stark difference between renters and owners in how people treat the units and common areas, treat their neighbors, and the level of involvement and care they have for the complex itself. Certainly some owners create problems, but having a vested interest makes a big difference vs. certain renters who could give a shit. And, problematically, when the owners move away and decide to rent, they usually stop giving a shit too.