How is allowing current owners to assert control (via regulation) over land they don't own, to serve their own private interests, better than allowing that land to be developed for higher-density use, in the interest of a larger number people?
Society isn't obligated to protect people's investments at the expense of its least fortunate members.
1) Why is using supply / demand the best way to set a price for a house?
2) Why should Americans be able to live and protect a town and prevent it form becoming a big city?
As for 1, because that is human nature. Any other method that goes against human nature doesn't work in large scale. In places that set a price, typically products are given to people with connection (see Soviet butcher getting their kid into a good school by giving the best cut of the meat to the teacher).
With regard to 2, this point comes down to what rights you think someone should have. I believe that a community should be able to form and develop themselves without too much government interference. Let's say that a community builds themselves under a tight bond, like the Amish. Imagine if the government wanted to create a huge homeless shelter in their community and house many previously homeless people there. That would completely disrupt their peaceful way of life. It seams logical to say that in that case it is bad what the government did. Similarly, some people building a town don't want the crime / dirtiness / corruption of big city. If a politician were to campaign that they will turn every town into a big city, I doubt that they would win.
With regard to topic 2, no community has the legal right to exclude anyone from moving there. If it makes economic sense for newcomers to arrive, they will arrive. If supply and demand were allowed to operate, homes would be built for them as they arrived. Instead the wealthiest members of the community interfere with supply, for their own benefit, and the least fortunate members of the community are evicted.
(I'm disregarding the homeless shelter in the Amish community because it's a contrived example. It's too far removed from any actual set of conflicting interests to have much bearing.)