Diesel 2: Using generators for async (and back again) | Hacker News Reader

Diesel 2: Using generators for async (and back again)

devblog.bu.mp

55 pointsjmintz14 comments

Does anyone else see a trend in this migration to async style programming? I've been working on a similar issue for a while now involving a similarity search engine with nested state machines.

I can't quite tell if my perception of this trend is skewed by my current perspective or if there are more people exploring this model of programming. IMHO, some things get much harder to reason about without some good tools.

I think the move to async frameworks is primarily precipitated by the fact that a file handle (which is, essentially, what an async framework holds per connection) is substantially lighter weight than a thread. When you're talking about writing a server that can handle thousands of concurrent connections, threads just become computationally infeasible.

That depends a lot on your OS and your threading mechanism. It used to be pretty bad on Linux for example, but the modern NPTL implementation is much better. And even then it still doesn't mean that your language has to stick to a 1:1 threading model.

The main problem with threads is the relative complexity of the programming and debugging model. I don't really see a complete migration from threads to async, I see a migratino from threads to a multitude of models, where async is just one of them.

Yes, that makes sense.

In my case, I ended up here by trying to dynamically shift where I'm applying the work load in the search space without pruning off an answer.

I'm curious if PEP 380 (yield from) would make greenlet less convenient.

http://www.python.org/dev/peps/pep-0380/

Yeah, PEP 380 definitely adds some more heft to generators-as-threads, removing a fair number of the problems I cited in that post.

Which remain?

The advantage of generators as threads is that they work with other Python implementations like Jython and IronPython. It would be nice if whatever Python adopted in the core wasn't still inferior to greenlet based libraries.

Deleted comment

> This looks a lot like monocle (Python) and async.js (JavaScript), which do the same thing.

Actually Diesel used to be more like monocle using yields. But the point of the article was that they re-wrote Diesel to use greenlet in Diesel 2. It is a rather different approach. Yes, underneeth they all have a select/poll/epoll event loop but greenlet is a lot cleaner approach for some use cases.

I use eventlet (http://eventlet.net/) which sits on top of greenlet. So far, I am very satisfied with it. I moved to it from Twisted. It felt like moving from Java back to Python.

You should also check out gevent (http://gevent.org) which is similar to eventlet.

Thanks Denis gevent is sweet, I monitor the project and might switch to it one day.

As of now I still use some of the eventlet-only features like thread pools and subprocess handling. Performance has not been a problem yet as well. I imagine gevent is faster since it sits on top of libevent ...

There are contributions on the mailing list of threadpool and subprocess handling. But you're right this should be in the library - we're working on it.

Congratulations, you've just re-invented Eventlet.

"Now with 100% less crazy socket monkey patching!"

Reply on news.ycombinator.com