Python co-routines: follow-up to Python generators presentation [pdf] | Hacker News Reader

Python co-routines: follow-up to Python generators presentation [pdf]

dabeaz.com

134 pointsirahul14 comments

Python generator tricks is garnering attention http://news.ycombinator.com/item?id=3349429

This is a follow-up presentation.

I would advice against using raw co-routines for co-operative multitasking though, barring a few special cases; gevent/greenlet will be a better option.

In section 9.1 and 9.2 of Programming in Lua, http://www.lua.org/pil/9.2.html, there are some explanations of coroutines, and the example of consumers and producers. Also there is a comparison with pipes, a key point is: "The cost of switching between processes is high. With coroutines, the cost of switching between tasks is much smaller (roughly the same cost of a function call), so the writer and the reader can go hand in hand.

I'm observing Python language elements being rediscovered on HN recently. These things have been in use for a while now and I'm wondering why they are getting so much attention several years after they were first introduced.

Is this because you are new to Python (or programming in general) or because these concepts and use cases were never explained properly, or you simply like Python and up-vote related stories?

(I'm actually asking myself similar questions about PostgreSQL stories as well.)

Anyway, I encourage you to take a look at http://docs.python.org/whatsnew/index.html (and http://docs.python.org/dev/whatsnew/index.html) from time to time.

It's not specific to Python, it seems to happen in every topic. On a related note, it would be nice to finally see that new posts of the same link would be tied to the other instances (or rejected).

The same URL can't be submitted twice. If you submit a dupe, it just gives a point to the original. Note that for the generator tricks submission, the submitter put a "?" after the URL to get it past the dupe-checker. Other tricks people play are to link to different parts of the site (say, the page containing a presentation instead of the presentation itself).

The "generator tricks for systems programmers" presentation has been on HN several times. It's one of those links that seems to pop up about once a year. It's not necessarily that people are learning about them for the first time, but they think, "Hey, cool, let's talk about this."

The problem with the Python documentation is that it does not adequately explain the value of it's more obscure features (such as coroutines). And since coroutines are obscure in general, programmers won't even know how to appreciate them once they find them (say in PEP-0342 http://www.python.org/dev/peps/pep-0342/).

Posting articles such as this one is very helpful because it pushes useful esoteric knowledge into our consciousness. HN is particularly suited because the community's up-votes convince us that our time will be well spent looking into such articles.

I find generators powerful and I use them all the time, but I haven't found a use for coroutines in my code yet. Maybe I don't grok them fully.

I feel the same way. I've found excuses to use relatively 'obscure' Python features (e.g. metaclasses) in the past, but for some reason coroutines have never seemed like the right solution to any problem I have faced. This might be because I don't fully understand them, or perhaps I am using them all the time and have no idea. But if that's the case I think that means they are either too difficult a concept to be of practical use, or an unintuitive way of thinking about what they are actually doing.

Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do.

Greenlet is an optimized implementation of coroutines for Python. Gevent or Eventlet combine greenlet + a scheduler and even monkeypatch your blocking calls into nonblocking ones.

I use eventlet for my load testing tool, Micro Army, to create nonblocking SSH pools. I can SSH to 2 boxes and run config scripts in parallel in 58 seconds, but I can SSH to 100 boxes and run the same tasks in 106 seconds.

The code couldn't be simpler too. Here's the proof.

SSH details for a single host: https://github.com/j2labs/microarmy/blob/master/microarmy/fi...

SSH to every host in a list: https://github.com/j2labs/microarmy/blob/master/microarmy/fi...

Notice that the code is basically just spawning a coroutine and iterating across a list of coroutines. Done. No callback spaghetti, yet all of this is async and nonblocking.

Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?

It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.

ditto for "coroutine + scheduler as replacement of threads".

Another example using this framework is Simpy (http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive!

The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.

I'm surprised how much he accomplishes without the ability to yield from a nested call.

Reply on news.ycombinator.com