If you are not, then you are probably not making the right comparison. Supersonic travel will absolutely not be for those who are flying economy right now and complaining about reclining, legroom, and crying babies. The kind of things that make flights feel very long, and yet, that's how most people fly despite much superior alternatives, because it is cheaper.
Judging by how much it cost to fly on Concorde, it is reasonable to assume that a supersonic ticket will be equivalent in price to first class, or at least a very good business class.
It means a seat that can recline 180°, good enough to sleep on, an internet connection suitable for remote work, as I expect it to become standard in the near future, a decent meal and some privacy. In these conditions, saving a few hours may not be as desirable as it is in economy class. Knowing that in order to shorten your trip, you will be sacrificing some of that comfort, or pay even more, maybe getting close to private jet territory.