Gevent, Long-Polling & You

Instead of "long polling" (or a different way to have long polling) is to use the HTML5's server sent events (http://www.w3.org/TR/eventsource/) See here for examples: http://www.html5rocks.com/en/tutorials/eventsource/basics/

I have implemented such a functionality for real time monitoring using gevent and works extremely well. This HTML5 feature is supported in recent versions of Chrome, Opera, and Safari and in the beta versions of Firefox, but not in any version of IE: http://caniuse.com/#feat=eventsource

Great article. I'd classify this approach as "streaming" rather than "long polling" though - in long polling, a browser opens a connection to the site which stays open until an event occurs, at which point it closes. As far as the browser is concerned, the request just took a few seconds to return - so the technique often bypasses dubious proxies and other issues (browsers have been really good at dealing with slow loading connections since the modem era).

Streaming is when the connection stays open and multiple messages are sent down the single pipe. It's faster and more responsive (and easier to scale since you don't have to deal with lots of connections being closed and reopened all the time) but is tricker to get working.

Of course, WebSockets make everything work sensibly hence avoiding the need for crazy hacks.

Here's a deck from a talk I gave on Comet back in 2008 which touches on some of this stuff http://lanyrd.com/2008/skillswap-goes-javascript/skq/

> [...] wondering about the yield ' ' * 1200 in there. [...] some browsers (like Chrome) will sit & buffer if there isn't enough data to start with [...]

Slightly off topic, but I wonder if this technique can be used to circumvent the buffering that occurs in HTTP proxies? I encountered the problem once and was only able to make the application responsive by closing the connection after a timeout (forcing the client to open another one).

Great writeup. This shows the power of gevent+redis really succinctly.

I think most importantly vs a twisted or tornado based solution this is using the vanilla redis client library. As long as your datastore's client library is pure-python it JustWorks with gevent.

Adding gevent-websocket and gevent-socketio to the mix gives some interesting possibilities as well.

Tornado (http://www.tornadoweb.org/) can deliver long-polling with a pretty small amount of work as well, and I appreciate the simple, lightweight design of the framework itself.

I went ahead and ported the pusher to Twisted, as a simple exercise; see https://github.com/MostAwesomeDude/wsgi_longpolling/blob/mas... for code.

Edit: And an explanatory blog post is at http://corbinsimpson.com/entry/summertime for those readers not well-versed in Twisted.