> only to stop 4-5 times along the way

That's the big one. What HSR needs is not even higher top speed but some control breakthrough that allows much higher train frequencies in order to cut down on stops via more trains that can serve more specialized connections (e.g. the local/through etc). No idea what that breakthrough might be, perhaps something as crazy as rocket-assisted emergency braking.