It's just that many of the stakeholders in these areas such as long-time property owners in the westside of San Francisco and residents with rent control fearing gentrification would not benefit directly from new development. Their incentives simply aren't aligned with younger generations who want to live and raise families in a city with reasonable housing costs. To break this quandary we simply to need intervention at the higher level. Congestion, long commutes and sprawl is hurting the U.S. (and Californian) economy so when change comes it will come at the higher level than the city.