people don't mind the experience of flying in a plane or the time it takes for the most part - they mind being uncomfortably crammed into a seat for hours on end with another person spilling into their lap in a loud, stuffy cabin. otherwise, it's just hanging out in a different place than you usually do.
at the point you're paying for a resort hotel room with a shower, bed, privacy, internet and a tv in the air... who cares if you spend a few extra hours? the only example of a supersonic airliner that i can point to, the concorde, was actually fairly uncomfortable and cramped because of the way it was designed. it's likely (though i've been wrong before) that future supersonic planes would make similar tradeoffs to try and minimize weight and drag and maximize fuel economy - you will trade comfort for speed.
i think most of the people you're talking about would prefer 8 hours in a private hotel room (or full on private jet) with a full bar, bottle service, a shower and fancy meals to 2-3 hours cramped in a relatively small cabin after the novelty wears off. given how much easier it is to effectively meet across the ocean without traveling, the market for ultra-fast flights to get a one-day trip over with is also likely smaller.
i can't say i know any of these facts for certain, but previously when discussing the return of supersonic flights with folks who know better than i, this was the general sentiment. it makes reasonable sense to me on its face.