Governance and land use decisions definitely play a big role in how the Bay Area has become messy. The Bay Area doesn't have a regional transit authority empowered with the final word on routes and infrastructure, it has at least one agency per county, plus additional ones for each railway - dozens of services that have difficulty coordinating.
The anti-development climate also played a big role in how things played out. Regional planning in the midcentury wanted something more like LA - lots of freeways, cheap land and cheap development. Public backlash prevented the bay from being filled, and stopped most new freeway construction. [0] BART planning started in the same era, and it was also more ambitious than what ultimately came to pass. [1] Yet despite that, or maybe in part because of it, the jobs, and subsequently new population, continued to stream in. As a result, there's a lot of "development debt" that has been kicked down the road until now.
The governance issue and the land use issue are correlated; since authority is heavily localized across many departments, even on basics like water(SF enjoys a direct line to the good Hetch Hetchy water), every city and county tries to avoid "taking one for the team" and focuses on its immediate self-interest instead, favoring NIMBY policies, unbalanced commercial development, and services and infrastructure favoring the wealthiest demographics. Governance is also generally inefficient; SF's spending is far beyond other cities of its size. The new Bay Bridge span is an embarrassment on many levels. Politicians regularly play the liberal base for fools by using a framing of concern to avoid useful analysis, decision, or action; at the same time, they are willing to bend over backwards for the VC-backed tech club, cutting special deals and ignoring or downplaying misbehavior.
It's a highly dysfunctional place, albeit not yet melting down.
[0] http://www.sfgate.com/bayarea/article/In-1959-the-vision-for...
[1] https://www.reddit.com/r/bayarea/comments/zprpe/the_actual_o...