The true nail in the coffin was the development of the lie flat business seat, which meant that you could cross the Atlantic in three hours in a plush but cramped seat, or spend less money to sleep for six on a redeye and arrive well rested. At that point three hours was not a compelling enough time savings, but the Concorde also didn't fly far enough to do routes where the speed resulted in more significant time savings, like on transpacific routes.
Always quite busy, and personally I’d much rather not try to get a good night’s sleep on a barely 6 hour flight.
Two-thirds of BA's Concorde regular passengers died on 9/11. The service never recovered financially from that loss.
Bannister knows probably more than anyone about the topic and tells the story well. Thoroughly recommend the book. His tale of a guy called Bill being invited into the cockpit and discreetly given the controls to fly supersonic was awesome. Of course, later over a beer Bill (Weaver) talked of his times flying the Blackbird at twice the speed, and of the time it disintegrated around him.