That Merced to Bakersfield line is characterizes as from nowhere to nowhere in How Big Projects Get Done. It’s one of the few negative examples in the book that stuck with me. It’s embarrassing.
Then again, at about $45 million per mile, it's actually not even that much more expensive that Chinese HSR: to build the current 45000km of rail at the LV-LA price would be $1.25 trillion, which isn't vastly more than the Chinese system has cost so far. In fact it's very roughly about as much as the US spends, per year, on healthcare on top of what European countries pay for the same outcomes (total annual spending: 4.2 trillion and a bit, or 12.5k per person, vs global number two, Switzerland, at 8k: about 1/3, not far from 1.25/4.2!)
Those will be built as (if) they are funded. The only funding currently secured is the bakersfield to merced leg.
The problem is, that leg is nowhere to nowhere and is going to cost over 100B dollars. Most people take that to mean that it's incredibly unlikely that the more costly portions of the route (the ones that make it useful) will ever get built as they will be absurdly expensive.
I know that there are many factors affecting progress, but it's depressing that we can't get to consensus on building infrastructure.
Connecting them all up with rails and catenary is the (relatively) cheap, quick and easy bit, and it's done at the end - it's exactly all these structures (and the stations, which will likely be the next phase) that take all the time and money.