Was the problem that supersonic flight was expensive and the amount of customers willing to pay the price was even lower than the number of customers that could even if they wanted to?
- They were loud (sonic booms were nasty).
- They were expensive to maintain and operate. Guzzlers. (Britain and France clung to them as a matter of pride/ego)
- They were narrow and uncomfortable. I have seen videos where there is space only for one stewardess to walk. I had been inside of one in Seattle museum. Very cramped.
- As you mentioned, ticket cost was high.
- I suspect people traveled in these mostly for bragging rights.Contrast that with modern widebody jets, which fly ~300 people plus paid cargo on much more fuel-efficient engines.