NIMBY.
The rich folks on the west side of the bay don't want extra train traffic through their enclaves. They like it gridlocked, thank you very much, so that the plebians have to stay out. And they have enough money to fight it.
East bay, on the other hand, has been working on it. BART is apparently almost to Milpitas.
Also, most of the activity nodes of the Bay Area are actually close to transit, which paradoxically hurts the efficiently of the system. Traditionally, you'd want each station to be its own hub from a larger catchment area that you'd serve with, say, buses or other transit. But in the Bay Area, because of the Bay and mountains hemming in the built-up areas, every interesting node is already on the One True Circum-Bay String of Pearls, and have minimal catchment areas beyond them.
This also means there is no express option to get from one end to the other, without having to traverse through all intermediate points first, regardless whether that's accomplished with an 'express service' that skips stops, or a transversal direct line.
[1] https://upload.wikimedia.org/wikipedia/commons/0/0b/SF_Bay_R...
The anti-development climate also played a big role in how things played out. Regional planning in the midcentury wanted something more like LA - lots of freeways, cheap land and cheap development. Public backlash prevented the bay from being filled, and stopped most new freeway construction. [0] BART planning started in the same era, and it was also more ambitious than what ultimately came to pass. [1] Yet despite that, or maybe in part because of it, the jobs, and subsequently new population, continued to stream in. As a result, there's a lot of "development debt" that has been kicked down the road until now.
The governance issue and the land use issue are correlated; since authority is heavily localized across many departments, even on basics like water(SF enjoys a direct line to the good Hetch Hetchy water), every city and county tries to avoid "taking one for the team" and focuses on its immediate self-interest instead, favoring NIMBY policies, unbalanced commercial development, and services and infrastructure favoring the wealthiest demographics. Governance is also generally inefficient; SF's spending is far beyond other cities of its size. The new Bay Bridge span is an embarrassment on many levels. Politicians regularly play the liberal base for fools by using a framing of concern to avoid useful analysis, decision, or action; at the same time, they are willing to bend over backwards for the VC-backed tech club, cutting special deals and ignoring or downplaying misbehavior.
It's a highly dysfunctional place, albeit not yet melting down.
[0] http://www.sfgate.com/bayarea/article/In-1959-the-vision-for...
[1] https://www.reddit.com/r/bayarea/comments/zprpe/the_actual_o...