Twisted.web vs Tornado Performance, part deux | Hacker News Reader

Twisted.web vs Tornado Performance, part deux

apparatusproject.org

52 pointsroder35 comments

I question the relevance of these tests. With all the attention being given to performance, everyone seems to be overlooking the fact that Tornado provides zero support for truly asynchronous web applications written for it.

If you have any kind of long-running task to perform during a web request, you have to roll your own way of dealing with that task.

To make it worse, the demos included with tornado that make use of a relational database do all their queries in blocking fashion, which would render the server completely unusable during the runtime of every single query.

Given the fact that Twisted provides a robust API for dealing with long-running algorithms, as well as support for a huge number of other protocols, you're paying an awful lot for only slightly better performance.

In fact, in a real-world application, the performance benefit of Tornado is directly impacted by how good you are at dealing with asynchronicity.

According to the Tornado documentation, FriendFeed runs many instances of the Tornado server. If the number of server instances is greater than the number concurrent requests, then there's no problem with blocking calls. I personally find synchronous code easier to write/debug/understand than Twisted deferred.

I think your point is really valid, though I think the relevancy of the tests were to test the claims made by Bret from tornado.

Glyph made a really good point, tornado comes with zero automated tests... Ultimately after conducting these tests and listening to the discussion that has taken place, I am personally glad that we're writing our code with twisted.web

What are you writing with twisted.web?

Stay tuned :) !!!

I will. Ping me if you get a moment.

Actually, I think their solution is pretty elegant; any task that takes that long is abstracted to its own HTTP request, which itself can be blocking without blocking the original request.

Summary:

  * Mean response time -- Twisted.Web better than Tornado

Average performances for Concurrent request:

  * @100  -- TW:Tor::0.50s:0.70s . TW better than Tor
  * @500  -- TW:TOr::   3s:3.5s  . TW slightly better than Tor
  * @1000 -- TW:Tor:: 5s:6.5s    . TW better than Tor

Thanks, this is helpful. Not to be a stickler, but two things on your summary:

* Where are you getting your estimates from? The numbers on the spreadsheet in the article do not match your rounded summaries.

* Twisted is faster at the same proportions for all the concurrent request numbers. Your summary makes it sounds like it's only slightly better at 500, but significantly better at all other request numbers. In fact, it's about 120% faster across the board.

Since yours is the top rated comment, I think you should ensure that it's accurate.

This comparison makes me wonder what the upper limit is for python-based server performance. I wonder where the main bottleneck lies.

Executive summary:

Ignore this "benchmark", the author simply doesn't have a remote clue what he is doing.

500ms Mean Response Time at 100 concurrent requests, are you kidding? Did you run the "benchmark" over a 56k modem?

For the record, one would expect sub-100ms response times under that load, even on modest hardware.

wow...

The test was run on two VMs on the Rackspace cloud using the public IP addresses.

I still don't understand all of the fuss. These two are close enough in performance to where people should pick the framework that best serves their needs, their designs, and their brains.

Big deal -- there are now two ways (at least) to accomplish the same outcome in Python. Competition is good. I'm guessing that this will be a call to arms for the Twisted community and we'll see development accelerate.

At the same time, it is likely that a development community will form around Tornado. Great!

It's like people saying, when the Mac OS first launched, "Why didn't they just use DOS --it's there and it works." And yet the Mac OS has had a positive impact on operating systems produced by Microsoft. And Microsoft has impacted the Mac OS. It's all good.

And why is it that this same argument doesn't occur every time a non-async framework appears for python or other languages? What is it about the async nature that polarizes people? Maybe it's the fact that the both also start with the letter "T" and the name "Tornado" is a better named than "Twisted" (which implies increased complexity even if it isn't there...of course Tornados ultimately make a mess of things, so that's not it).

I, personally, am excited about Tornado. I think there's room for more than one player -- especially given the 'real time' direction the web is heading. And if this negatively impacts Twisted, then it'll be because Tornado has come up with an easier to understand approach and they'll benefit from being new and easy to grok -- in the same way Rails was once an easy-to-grok framework and that's what made it easy to understand.

The biggest winner, in my opinion, is the Python community and who doesn't want that?

I don't think these tests are much more meaningful than the first.

They still use virtual machines, which give you all kinds of invisible overhead.

Even if that were not the case you should definitely split the machine that does the tests off from the machine that is being tested otherwise you get contention between the test program and the programs being tested (and java being quite cpu intensive running both tests on the same physical box is not a good idea).

Given how common it is for production applications to be running in the cloud, this test is both valid and useful. Therefore these tests are much more meaningful than the first.

I disagree. If you run a test on a virtual machine then you are testing a lot of other software besides the software under test so the picture gets distorted.

For instance, in the graphs that are in the report you can see a bunch of drops to '0' traffic, these suggest serious problems with the software under test. But knowing that the tests were done in a VM environment it is possible that the cause of these drops is not in the software under test but with the VM, or alternatively, with other software running on the host the VM is running on.

That makes it very difficult to assign meaning to all this.

If you test software you do it in an environment that controls all the variables as much as possible otherwise the results will not help you in making decisions.

If people run software 'in the cloud' then that's a different story altogether, then you test the same software side-by-side on your bare machine vs your 'in the cloud' setup and you benchmark those.

This test is labelled 'twister vs tornado', not 'twister in the cloud' vs 'tornado in the cloud'.

If the tests were all over the map, I could agree with your original statement. But they are not. They are consistent.

Do you honestly expect anyone to believe that when test after test is run on a VM, and one framework consistently outperforms another, that this is due to the VM, and not the frameworks themselves?

I'm more concerned with the drops to 0 and the overall throughput. The drops to 0 look quite bad, overall throughput is low compared to what I would expect from a setup like this.

But if you're testing them both in a VM, wouldn't you consider this a valid performance benchmark for those looking to run their instance in a cloud VM?

I mean, I think I get your point, in that this benchmark doesn't apply to those running bare metal... but isn't the obvious counterpoint that this is an especially useful benchmark for those running in a VM?

If you compare two things running on a VM you have to have a play-by-play idea of what else is running on the machine the VM is running on otherwise your test could be seriously affected by something outside your view.

I did split the machines. The driver (the one running the test) machine is completely different than the webserver running twisted.web or tornado.

But do you have any idea what else was running on those virtual machines and what kind of overhead that gave ?

Yes, I know exactly what was running on those virtual machines. They were purpose built for this test, nothing else was creating load on those servers.

The virtual machines were not running anything else, but the machines they were running on might have been running something else.

Or are you sure that the machines these virtual machines were running on were otherwise completely idle ?

And if they were idle then why bother with the virtual machines, then you could have just run your tests directly bypassing the whole VM overhead.

Also, they're testing Hello World. IMHO that makes the tests close to meaningless.

The reason why I used "hello world" is two-fold

#1. because it's what Bret from Friendfeed used in his testing #2. it tests the core framework code, anything more is testing code I write.

I guess i will never understand why they didn't choose twisted and finally gave it the revamped twisted.web it needs.. I have yet to see a more featurecomplete internet framework with which i can do practically everything the internet is about, with so much flexibility and yet it is suffering from the state of twisted.web. Aynchronous database access, http, pop3,imap,smtp, amqp, irc, msn and about every protocol out there can be so easily integrated in twisted, it amazes me everytime. Sad, that this framework isn't used more widely, it could rule the web.

Innocently daft question coming up:

As I understand, you can run web.py apps behind lighttpd, and the web server handles the processes. E.g. in lighttpd.cfg you can set {"max-procs" => 4} and I would see 4 instances of my web.py script running. But how is this done for a web-server done 100% in python? Python uses a GIL that only allows one thread to interpret bytecode at a time, so despite the amazing performance of Twisted and Tornado, wouldn't we see this seriously degrade when the server had to anything more than 'hello world'? My worry is having a Twisted/Tornado app and only using 1/4th of my availble processing power: am I wrong to do so?

That applies to threading.

Twisted and Tornado are epoll/kqueue/etc-based and do not use threads.

Here's a good article on the differences between strategies: http://www.kegel.com/c10k.html

Great article, thankyou!

See http://www.tornadoweb.org/documentation#running-tornado-in-p... . FriendFeed uses nginx as a reverse proxy to run several instances of the Tornado web server process. Each process has its own GIL.

Thanks, I'll give it a go with lighttpd too.

These tests are completely useless, since most of the measurements are done at a point where response time has already degraded unacceptably. I don't need a web server that performs slightly better when massively overloaded.

The sweet spot to measure is between a response time of 0s to 1s. After that, you need to be looking at getting more servers, anyways.

A better benchmark would start by swapping the axes of the graph and asking the question: "how many users can I support while keeping response time acceptably low?"

I would really like to see Twisted vs Tornado Documentation. The performances are close enough that I don't think it should be a deciding factor in which of these to use. Albeit performace is always an important metric that should be watched to keep things from getting worse.

In fact, all those comparisons compare two totally different frameworks. One is for the internet (which really includes far more then http) and one is for the web (which mostsly includes http).

Reply on news.ycombinator.com