The housing crisis is perpetuated by a lack of coherent planning. It's also why transit is suboptimal. It definitely doesn't help that it isn't one big city, it's actively detrimental.
Even travel down the Peninsula is extended because of the linearity of the Peninsula (you travel up and down it).
It makes more sense for there to be regional centers. Develop Oakland. Develop San Jose even more. Maybe turn Hayward into a regional hub (fat chance but the geography is interesting). San Mateo is growing for a reason.
Hook the centers together with fast regional point to point transit. There should be no stop trains between SF, San Mateo, San Jose, Oakland, and Hayward.
It is all incredibly obvious if you look at a map. It is all driven by geography.
This is also the reason that Mountain View and Fremont have grown rapidly. Just look at the map. Maybe add them as regional centers.
You don't need a large city to coordinate transportation. By that logic, the whole Bay Area should be one city.
After all, you have BART running to the East Bay, South Bay, and a tiny part of the Peninsula.
The proximal problem is that the SF government is too inefficient and slow. BART is actually more efficient than SF is at building stations.
Why would you want drag more SF politics into something that runs better than it?