The free market also doesn't lead to more getting build in locations where no space is left. Why are waiting times worse than extreme prices?
Holding pricess below market rates helps ensure that over time current housing will be less maintained as money goes elsewhere, and as current housing can be converted into other more profitable land, it will be, thereby reducing housing overall.
There are 100s of papers showing this, which is why rent controls being bad is one of the things most economists agree on.
Higher prices = more incentive to build. It's that simple.
There's plenty of space left in places like New York or San Francisco, especially vertical space. Whether that's legally available to use (or impacts local infrastructure) is another matter, naturally.
Because of the zoning laws - the opposite of a free market.