Completely agreed. I mainly fly internationally, and while shaving 2 hours off a 8 hour flight would be wonderful, it doesn't help me, as the majority of my time is spent in airports between the flights.
For me to get to where my family lives (one layover), it takes me:
1 Hour drive to airport
2 Hours because it's an international flight
8 Hour Flight
2 Hours Immigration / Baggage
1 Hour Re-checkin
3-5 Hour Layover
30 Minute Commuter Flight
30 Minutes for baggage
30 Minutes drive home
So for my entire flight, the flying (even on a 8 hour flight) is less than half of my total time. By lowering the amount of time between connections, or lessening the amount of connections that need to be made, you can get greater gains than by flying faster planes.
Cheaper planes and cheaper flights could actually make overall flying time SHORTER, because there would be more planes in the sky, and less time between fights.
-----------------------
Long rambling story:
When I first rode the bullet train back in college, I thought it would be like riding an airplane, at least from a management perspective. My friends and I bought the ticket a week in advance, made sure we were packed, and got to the station a good hour in advance of my train leaving.
It was only then that I realized that the security for getting on a bullet train is "show me your ticket" and that if you miss your train, you can just sit in the non-reserved seating on the next available train that will be along in under 5 minutes. (Back then it was more like 10, but still)
While you can't really compare trains and planes (and automobiles) in these situations, it still makes me think that the solution to faster travel is not faster planes, but better infrastructure and administration for travel.