I think people are getting distracted by the idea of connections being somehow expensive. They aren't really compared to polling (unless the poll is genuinely very rare). A stateless request is expensive because you have to go back to your source of truth on every request (probably an expensive and hard to scale RDBMS), and you don't control how often the user makes such requests. CPU load is potentially unbounded and users don't pay unless you introduce pay-per-poll micropayments.
Compare that to an MQ design: the overhead is a single TCP connection and a bit of memory to map that connection to an internal queue. Whilst the work sits in the queue or is being processed, nothing is happening and there's no DB load. Overhead is a matter of bytes and in the event that you run out of RAM you can always kick users off at random and let them exponentially back off and retry (automatically - because the libraries handle this and make it transparent). Or just use swap, after all, latency is not that important.
So, you just can't rely on your customers being able to keep around a long connection.
Not to mention the numerous corporate environments in which it is hard to even open an outgoing connection which is not HTTPS or a handful of other known protocols.
As for the general issue of connections, that's true for consumer use cases. B2B workloads have far fewer problems with that especially when running in the cloud. If your cloud gives you mobile-quality internet then you have a problem, but again, it's a problem a good MQ implementation will fix for you. Consider the "lessons from 500 million tokens" blog post the other day, in which the author mentioned repeatedly that they had to write their own try/catch/retry loops around every OpenAI call because their HTTP API was so flaky.
And again, if you are behind a nasty firewall then you might find your connection dying at any moment because OpenAI got classified as a hate speech site or something. The fix is to file the right tickets to get your environment set up correctly.
This is a very pessimal case, though. You make a tiny HTTP request which is parsed in microseconds at the most, and it invokes somewhere between one to five million microseconds of 100% utilization of an expensive resource. A thousand queuings per second would be fairly easy for a RPi, it could handle that no problem (at least assuming you use a decent language to manage it; a super fancy Python framework that also does a lot of metaprogramming might choke under the load, sure, though some half-carefully written Python still ought to be able to handle this), while those 1000 requests/second would require around 2000 GPUs to dispatch them in real time so we can maintain that 1000 rps. I'm pretty sure you can add an order of magnitude before I'd really start worrying about the RPi as a queuing mechanism, and you're getting to the RPi being able to queue for ~100,000 GPUs without too much strain. I don't know how many GPUs OpenAI has but that's got to be getting pretty close to their order of magnitude. They may have a million, I doubt ten million.
(Of course, I wouldn't actually do this on an RPi; I'm just using that as a performance benchmark.)
https://dropbox.tech/developers/low-latency-notification-of-...
This open connection IS the overhead. Some approaches have more overhead, some less. Long-polling these days is not needed and can be replaced by SSE [1] (if you are OK with unidirectional communication), websockets (bidirectional), or callback URIs in the request. The latter is a per-request webhook, essentially, and would have the lowest overhead at the cost of ceremony (the client now needs to have a running web server).
By the way, long polling was popularized by Comet [2] around 2006.
It's not really easier to implement or scale, in my view. It may seem that way if you've never worked on large scale stateful connection serving. If you want users to get the answer as soon as it's ready, and you do or at least your users do, then you need users to hold open connections and at that point it's really a question of how much your protocols and libraries do for you. If you use HTTP the answer is "not much" because it was designed to download hypertext. The actual task of managing lots of connections server side isn't especially hard. There are MQ engines that support sharding and can have a regular TCP LB like a NetScaler stuck in front of them, or of course, you can just implement client-side LB as well.
The alternative with polling rapidly turns into being melted all the time - it's literally handling constant reconnection load - especially as you don't control the polling code and as the user is expected to write it themselves you can safely bet it will be MVP. Retries if they occur at all will be in a tight loop.
I worked on Gmail for a few years and we had a ton of clients permanently connected for IMAP, web long polling and Talk. Those clients tended to be well written and it wasn't the big problem people seem to think it is, especially as some could do client side LB which reduced resource requirements server side. I also experienced (luckily from a distance) fully HTTP polling based systems where backoff/retry wasn't implemented properly and caused services to go hard down because they couldn't handle being hit with all the polling simultaneously, and the clients just broke or did something stupid like immediately retry if they got server errors.
Fundamentally, state is fundamental and can't be removed, so it's going to be sitting somewhere. If state isn't in the protocol stack then it's in your database or session cache instead. Sometimes that's better, sometimes it's worse.