What about the land owner's will to build what they wish on their own land?

It's a greater kind of tyranny to monopolize land use and build walls to block the free association of people, and limit what a person can do with their own land.

If people want absolute control over the land in their town, they can buy the land and put a fence around it.