Edit: the Vermonter and Ethan Allen express have even less ridership but are shorter, I wonder how much money those lose.
The Northeast Corridor has at least hourly service through out the day, 'high-speed' (for america) trains, and lots of newish comfortable equipment, it's arguably the most civil way to get between NYC/DC/Boston.
A lot of the rest of the lines have once-per-day service, that runs on old equipment along freight lines. If you want to ride the Empire Builder from Fargo to Minneapolis, your only option is to get up at 2 o'clock in the morning to catch the one train of the day, and if that train had to wait for a freight train (which has priority) along it's previous 2 day journey from Seattle, it wouldn't be unusual for it be delayed until 3 or 5am.
Since the US doesn't invest in passenger rail outside the NEC, it becomes less and less viable for anyone to actually use it.
Anecdotally, I've used the Empire Builder for two trips this year, and both times every train was completely booked (on a weekday, no less). There is demand for these trains, which I suspect would be even higher if Amtrak ran faster, more frequent, and more consistently.
The fact that it is serving both these purposes is kind of a determent to both. It runs to infrequently (and at weird hours) to serve as a nice interurban, and it gets too overbooked to serve a tourist route. To fix this the state of Washington (and Minnesota) need to operate more frequent interurbans that only services the end portions of the route (like 4 trains a day at least), and then Amtrak could operate the whole Empire builder in a more sane manner.
This also means that there are very cheap fares on those trains within Vermont, and I've sometimes noticed that it's actually cheaper to buy a ticket from DC to a station in Vermont than to buy one to New Haven.