The real "agony" I find when flying is the faff of getting to or from the airport at weird times of day. From the airport onwards it's usually a pretty relaxed affair whatever the time of flight. Wish I had one button/dial to balance price against that faff!
Maybe it's just a London/big city thing?