Good luck finding anyone to live in that kind of neighbourhood by choice. If you have an area with zero controls, it quickly will become a race-to-the-bottom.
The kinds of people who want to live in a nice neighbourhood, and be nice neighbours, will go live somewhere else.
It turns out that people feel the same way about freedom to do what you like with your single-family home as they do about the freedom to drive a car without getting a license or being insured, and the freedom to drive on whichever side of the road you like, and the freedom to drive a vehicle without smog regulation or safety rules.
Most people don't want your freedom, because the upside of their freedom is greatly outweighed by the downsides of their neighbour's freedoms.
They'll all just go any set up gated communities where everyone buying a home must contractually agree to the neighbourhood's own regulations.