There are several factors at play:
- The bias against sufficient provisioning exists regardless of scale or growth rates. Housing tends to be underprovisioned when it serves as an asset, whether at the individual or institutional scales.
- Infinite expansion of housing stock isn't necessary. In mostt cases only a small fraction of existing housing stock need be added. California has a homeless population of roughly 170,000 (https://www.usich.gov/homelessness-statistics/ca/) on a given night.
That's a large number, yes. It's also less than 1/2 of one percent of the total population of the state.
The incremental housing need is small. It's localised. Those constraints of which you write will not be even remotely approached in addressing the housing needs of even 10x or 100x this number of people. Adding a small amount of additional housing at affordable rates in high-demand markets will address the housing problem nearly completely.
(There may also be some additional services for the addicted, mentally ill, disabled, elderly, and young. That's independent of the housing issue itself.)
But adding a few percent to the housing stock will have an immense impact on the very thin trade of that asset, plunging prices, and hence household and family wealth, as well as real estate, mortgage, bank, and financial derivative asset values. Given the utter dismanteling of defined-benefit pensions and the importance of real estate appreciation to family retirement planning, and the role of the financial sector in politics, the resultant political pressure is immense.
That covers your first two points.
For your fourth: yes, I fundamentally agree, that's what I just argued above, and it's a core aspect of the present homelessness crisis. Given that landlords can also drive evictions at will (effectively creating demand), the market is further unbalanced.
The geographic structure of the market is a non-issue when present low densities and low marginal change requirements (as discussed above) are factored in. SF Bay Area housing demand could be addressed with improved transit, increased densities, and stronger tenant protections with only modest increases in total housing stock.