Hacker News Reader

Comment by atdt | Hacker News Reader

I find generators powerful and I use them all the time, but I haven't found a use for coroutines in my code yet. Maybe I don't grok them fully.

View on HN

Coroutines + a scheduler become an awesome replacement for threads. They're lighter weight and don't thrash the GIL the same way Python threads do.

Greenlet is an optimized implementation of coroutines for Python. Gevent or Eventlet combine greenlet + a scheduler and even monkeypatch your blocking calls into nonblocking ones.

I use eventlet for my load testing tool, Micro Army, to create nonblocking SSH pools. I can SSH to 2 boxes and run config scripts in parallel in 58 seconds, but I can SSH to 100 boxes and run the same tasks in 106 seconds.

The code couldn't be simpler too. Here's the proof.

SSH details for a single host: https://github.com/j2labs/microarmy/blob/master/microarmy/fi...

SSH to every host in a list: https://github.com/j2labs/microarmy/blob/master/microarmy/fi...

Notice that the code is basically just spawning a coroutine and iterating across a list of coroutines. Done. No callback spaghetti, yet all of this is async and nonblocking.

Very interesting. Does Greenlet use threads behind the scenes? Do you run into the same problems with deadlocks, debugging, etc?

It's single-threaded. I haven't looked under the hood, but I would expect it to work by copying stacks around. In my experience it's been very easy to use.

ditto for "coroutine + scheduler as replacement of threads".

Another example using this framework is Simpy (http://simpy.sourceforge.net/), a process based discrete event simulation platform. You program in a way similar to the traditional multi-process/thread manner, but don't have to worry about mutex and locks at all, very sweet and productive!

The only caveat for this framework is that it's hard to fully utilize the power of multi-core systems, since it's single threaded. But if your application is more I/O bound than CPU bound, this framework could potentially give you huge benefit of multi-threading w/o the complexity.

I feel the same way. I've found excuses to use relatively 'obscure' Python features (e.g. metaclasses) in the past, but for some reason coroutines have never seemed like the right solution to any problem I have faced. This might be because I don't fully understand them, or perhaps I am using them all the time and have no idea. But if that's the case I think that means they are either too difficult a concept to be of practical use, or an unintuitive way of thinking about what they are actually doing.

Reply on news.ycombinator.com