Twisted Python's lead hacker discusses Tornado

I'm not familiar with the project or this guy in particular, but man, he seems awfully abrasive. Sarcastic and bitter came to mind as I read them. I have no dog in this fight, and one could fairly assume that these 2 discussions are just not his best moments. But from what I've seen...

Now that it's out in the wild, maybe someone will cross-pollinate, maybe changes will go in both projects. But the good thing is that they got their job done.

Ultimately, I think Twisted, due to the fact it does so much, is intimidating. The documentation lacks succinct examples of how to implement the very basic things people want to implement. The documentation to figure out how to make an app is buried under at least 3 clicks and even then isn't clear. The comments, when there are comments, in the examples don't truly explain the why of the actions just that they are there.

If twisted wants to be considered as a serious contender as foundation to develop web apps on, they need to take a look at how important very good documentation is to Django's success or how the simple, uncommented example at http://www.tornadoweb.org/ is more clear than the first bit of web example code at twisted: http://twistedmatrix.com/projects/web/documentation/examples....

I'm not knocking twisted, I'm trying to explain why when I went around evaluating what I would use in my projects, I did not choose Twisted. If you find my decisions and observations valid or not is up to you.

This is progress, and as a result, the Twisted people, if they're listening, have gotten valuable feedback out of the process from many people other than myself as a result.

There has also been much more about Twisted on places like Hacker News than I've seen, relatively speaking. With Dustin porting the meat of Tornado on top of Twisted's IO handler, there is much more interesting things to come out of it all.

It is in no way a negative thing for Twisted for there to be competition in an area of cross-over such as this.

On the contrary, my observation is based on comments I've read in many Python community forums, especially when applied to Python vs (your other high-level language of choice). For example, I've read many comments to the effect of "I like Python over (Ruby, Perl, etc) because it promotes one right way of doing things and having one package for handling problem X." I am not stating that this belief is a Python language requirement, nor do I believe that this observation adheres to all Python coders so yes, it is a generalization. But my generalization is certainly not based on a single observation. Finally, I will gladly play the devil's advocate and agree that there is little way to prove this generalization, short of spending several days collecting emails on Python mailing lists, which is why it is offered as an observation only. It is an observation of the practices of a community not something codified into a PEP, at least as far as I know based on my limited research.

I completely agree with the rest of your comment and value your critique. Python is a great language that provides a diverse set of tools, which is why I also choose it to handle many tasks.

That's not exactly what I'd say. :) Tornado without twisted is simpler than tornado with twisted, because you can't ignore the huge system you'd otherwise be able to avoid.

Edit to remove mild flame.

Or, looking at it another way, you removed Tornado's simple 1400-line core, and replaced it with a 150,000-line dependency.

(Yes, I know that Tornado On Twisted doesn't depend on the entire Twisted framework. But the port does involve a lot more code in total than the original.)

That seems a little strawman-ish, honestly. On the contrary, they embraced the technology and architecture of the app.

What's wrong with looking at something like this.. an important infrastructure piece... liking the idioms and architecture but having issues and thinking .... We have a benefit the Twisted guys didn't have... we have prior art... a map of this territory ?

I don't think anybody here has a strong conviction that FriendFeed was _right_. But you and others seem to have a strong convication that they were wrong, and all I think....and I think others here too.... all I think is that it is silly to attack them and be so certain they were wrong.

These are talented developers. Smart guys. With a little bit of business talent, too.. All of us have been in complex project cycles. Who are you or anybody to pick them off like they somehow caused somebody harm? Why the intensity? What gives?

Actually they don't. Twisted is "an event-driven networking engine written in Python"

Tornado is "an open source version of the scalable, non-blocking web server and tools that power FriendFeed"

Note that the scope of these things are completely different.

No, but twisted is much more complicated than regular python. The inversion of control and the fact that until quite recently it was almost impossible to write code for using the framework that was easy to read/understand by experienced python ciders who were not familiar with the idims and conventions used by the project. Twisted generates more WTFs per minute than any other bit of python kit. Some of this is due to a tradition of poor documentation within the project and some due to the nature of the code. Twisted is a complicated system, and sometimes this complexity is not worth embracing.

The author says communication could have cut their time down considerably. That may or may not be true. Getting into a debate with an external project about what should or shouldn't be the correct way to fix/change the external projects code can be a huge drain on time and mental energy. You can't fault a group for deciding not to bother.

Not to mention that if you become the person responsible for a piece of an external project you are suddenly not just supporting your use of that piece you are supporting everyone else's use of that piece. That leads to more debating/discussion about the right way to do it. If you have to find a way to scale fast it might actually in the long run be less time to just write it yourself for your specific needs not an entire communities needs who have different scaling issues than you and different goals. Many times One size does not fit all and convincing someone else of that is just too time consuming to try.

The author did bring up an interesting point about friendfeed developers not even trying to communicate though so I can understand what prompted him to write the article.

To me, this makes things more complicated. What if I want to use the same error handling code for all bits of code (really common for me is `d.addErrback(log.err)` to just log it), but each successful call has to do something different?

Twisted deferreds also allow you to compose chains.

This is a somewhat trivial example, but by having the callback do both success and error cases, I had to build an adaptor to pass to callbacks to include the "is this a failure" information.

The easy way to do that is to create a wrapper class and construct it one way or another based on whether a success or error occurred. I ended up just composing it like this:

    deferred.addCallback(lambda x: Response(body=x))
    deferred.addErrback(lambda e: Response(error=e))
    deferred.addCallback(lambda r: callback(r, *args, **kwargs))

This is sort of unrolling complexity and sweeping it away. The real problem is that you can only have one callback in tornado (and you can't easily chain them). That means that what actually happens when this deferred's callback fires is that it ends up calling the original callback, which is an inner wrapper function inside of another function that calls a curried function that calls your original callback -- and you have to create these things by name when you're building out your auth class.

To me, that's complicated.

deferreds seem like there's more involved than you get with a simple callback until you actually try to use the simple callback.

The interesting properties of deferreds are

1) You can add multiple callbacks and errbacks and build chains as complicated as your system needs including composing common cases (such as success and error counters).

2) If a callback (or errback) returns a deferred, you spin off another dimension of the chain.

#2 is why the above code and about twenty lines of web.py exist for a single use case.

Here's a concrete example (close to something I do in the real world™):

Imagine a simple twitter API. You go grab a tweet and then give it to something else: `t.grabTweet(something).addCallback(processTweet)`

Now someone adds a requirement that between `grabTweet` and `processTweet` you want to perform short URL expansion. In twisted, I'd just toss an `addCallback(expandURL)` in the middle and call my awesomely well-tested URL expander -- which makes an async http call to another service and doesn't call the next callback in the chain until that service returns the data I need to fill satisfy the request.

And then after I did that, I found that the URL expansion stuff I was using was blowing up the upstream and causing me to fail to expand some, so I added async cache support to those inside the API (not as an additional toplevel callback). For this new step, the toplevel API doesn't change, but now I can do more async stuff that may lead to further async stuff without having to write additional glue code.