SF needs more public transportation as much as it needs housing :(
Good transit is expensive. The people who already live in the area are not willing to pay enough to justify the investment. They have already figured out their lives without the transit. If you want transit, you have to build it primarily for new residents. And the costs should be covered by increased land value, usually either by taxing the land, selling it, or developing it. Trying to cover investment costs with fares just destroys the value of the investment.
Maybe one day we'll have fully automated buses without any operators at all, but for the time being my understanding is that even self-driving cars still need a tender. The kind of bus volume (and corresponding idle drivers) you'd need to make up for a train's worth of commuters probably takes a heavy cut into any operator labor savings.
I'm not aware of a city on Earth that's been able to obtain the kind of simultaneous density and livability that places like NYC have without mass transit. Comparatively dense cities without mass transit (like Manila) are choked in traffic; changing that traffic to electric traffic will make the air cleaner (good!) but will not address the latter problem.
That's mostly paying for the tunnels, not the track. It shouldn't cost nearly that much, but it's not surprising that it isn't cheap either: the tunnels are being constructed under some of the densest and most expensive real estate in the world, and the resulting stations are significantly larger than others in the system.
And note: if you think that's expensive, you won't believe what it costs to repair highways in NYC[1].
[1]: https://ny1.com/nyc/brooklyn/news/2020/02/25/city-council-fl...
Self-driving cars may make our highways more efficient, but they can't change the basic fact that a road with individual-sized cars on it carries far fewer people per square foot than mass transit does.
(And note: we currently subsidize the hell out of our road network, in a way that's far less obvious to the public mind than our comparatively light subsidization of mass transit. It remains to be seen whether the average car trip is actually net cheaper.)
I imagine that we'll always have roads, but rails systems are optional, so the cost of roads is a given. Widespread use of self-driving vehicles should dramatically reduce the amount of lanes that need to be maintained though.
I'm familiar with the arguments about rail capacity vs cars, but I live by main road and train lines. The road is in use 24 hours a day, and the rails are mainly idle. My dream scenario is self-driving vehicles become ubiquitous and we replace all commuter rails systems with foot and bike paths.
This is a measurement error: you might see 300 cars in an hour and only two trains, but those 300 cars are carrying 600 people while the trains are carrying 1500 each. The road appears busy and the rail system idle, but the exact opposite is true in terms of volume of people moved.
(It could also be true that the train system isn't well utilized in your area, but the fundamental point about scale and measurement is the same.)
As others have pointed out elsewhere in the thread, the main cost of mass transit is actually labor, not the rails (or even trains) themselves. The US is somewhat behind the times in realizing this; just about everywhere I've been in Europe has healthy mass transit systems that run cheaper and more reliably due to increased automation.