This is simple supply and demand. Everyone from liberal economist Paul Krugman [1] to SF board of supervisors member Scott Wiener [2] agrees that California and bay area housing policies are horrible. The only people who actually like these policies are the people who are benefiting from them, typically because they moved to a neighborhood decades ago and are either on rent control or "rent control for property tax" (aka Proposition 13).
The solution is simple. Set aside a square mile in each city on the peninsula and zone it such that it is illegal to build a building of less than 10 stories. At the same time, and more importantly, repeal both proposition 13 and rent control statutes.
For step 3, sit back and watch the California economy go into turbo drive as there is a massive construction boom, which funnels money away from rich landlords and into the pockets of ordinary construction workers and the service workers that support both them and the engineers the region's wealth is built on.
1 - http://www.nytimes.com/2000/06/07/opinion/reckonings-a-rent-... 2 - https://medium.com/@Scott_Wiener/yes-supply-demand-apply-to-...