It seems to me that the market will select for urban sprawl though which is a negative for society but has the highest margin. E.g. Houston suburbs, miles and miles of cheap to fab single family homes that turn it into a suburban hell scape where you have to drive everywhere.

I don't think the free market is giving the promises you say it is - supply isn't elastic for real estate if nobody's building because there's no margins. Demand can be anywhere really.

I like to look to Tokyo for an example. Small lots, extremely predictable regulations (that are still strict enough to ensure a safe living situation), fast approvals, mean it's much faster and easier to throw up an 8-10 story apartment than say downtown Austin, and so even today they keep doing it despite land in Tokyo being very expensive. And, no sprawl.