Its like saying the US should never have built rail across the US because most people don't use it.
Of course if you just do high speed rail, you don't have much, what you actually need is to extend and expand your regional rail and transit network. Then high speed rail benefits both regional transit networks.
Given how much money is invested in highways and how much it cost to build the absurd highway system, rail just makes way more sense.
Its pretty funny to me that what is supposed to be 4 largest economy in the world can't manage a half way decent train network. When there are multiple cities with 500k to millions people are between SF and LA.
There is easily enough population to have a really successful high speed rail if its correctly connected to a larger regional network.