A counterargument to this is that population growth has not been very high in the past few decades: https://fred.stlouisfed.org/graph/?g=SUL3
You certainly have valid points, especially in San Francisco. Nevertheless, looking at the broader market, outside of California, the market appears to be working: builders find a way to build what will sell and maximize their profits. Last week, I drove around a planned community in a middle-America state. Developers started building houses around ~2010 and are still building out the area. These developers had no problem building thousands of homes that were selling well until very recently.