For example, I rent a 400 sq foot apartment in SF for $3k. The operating and capital cost of the physical apartment is probably on the order of $1000 (I have rented similar places in Montreal, which has ample housing, for <$1000). That means that the other $2000, or about 2/3 of my rent, is the result of apartment scarcity caused by development restrictions. If new development is more costly, we could imagine my rent would be $1500 at market equilibrium if there were no barriers to development and the city was covered in high rises from bay to ocean, so I am effectively paying $1500/mo for restrictions on development.
Those restrictions have value in the form of reducing community impacts, so the question becomes, should the community as a whole pay a 100% "tax" on its rents to avoid additional development? The policy also closes off market rate housing in the city to any who can't afford the rents, which could be a feature or a bug depending on your outlook.
I believe there is a goldilocks regime between the two extremes that we should strive for, but our current system doesn't provide a mechanism for us as a society to make a balanced cost/benefit tradeoff.
Is there unanimous agreement that high rents(which are occurring throughout the US and abroad right now) are solely due to lack of development?
https://pbs.twimg.com/media/CgXjY3cWIAAVfCM.jpg:large
Unfortunately we can't run experiments on this question. Every city is unique, and people tend to think very special.
Is that where life becomes so unpleasant for the people who live there that the rents crash? I think you run into a traditional work migration problem here, because the young people who are moving to SF from suburbs and small towns across the nation generally have no intention of staying in SF and raising a family, so will be tolerant of far worse conditions than people who intend to actually make a life there.
Sparse housing has costs, too. Very big costs, like commute time and pollution.
The fact that a developer might get rich in the process of creating dense housing is not really a problem for me.