Oh yeah, you're absolutely right. I got a bit tangential. Slow establishment wouldn't matter for a stream. I'm sure it's something more like the e-commerce page load time vs conversions scenario for OP.
I could live with slow handshake, but it's ~110ms for every request.
I had a similar problem working with an managed file transfer service. Lots of small files would get bogged down by these sorts of issues. The other side of the connection wouldn’t change their config, so I implemented a proxy that maintained session state on our side.