As one example where it's already done, the DC-NYC Amtrak route is quite popular, despite being just as expensive, if not more so, as the plane ride. It takes a similar amount of time, factoring in travel to and from airports, waiting, etc. and is much nicer. There are still a ton of airline flights between these two cities as well, of course, and those could probably be supplanted to a large degree with additional improvements.
That's with current American trains, which are painfully slow. Imagine modern high-speed rail between, for example, DC and Miami. It's a bit over 1,000 miles which would take about five hours. That would be an excellent substitute for the plane trip, which is about two hours in the air, but tons of time and hassle at the airports. I have family in South Florida and if such a train existed at something like a reasonable price, I'd definitely go for it. As it is, Amtrak on that route takes nearly 24 hours and costs way more than an airline ticket. I don't see why they get any passengers at all, given that.
Seems like 1,000 miles is about the limit for high-speed trains. How many pairs of large American cities are less than 1,000 miles apart and have lots of travel between them? I would bet there are quite a number. Rail doesn't have to solve everything to solve a bunch of things.