Hacker News Reader

Comment by bhauer | Hacker News Reader

I worked with Pat (pfalls) on this effort. He pulled the benchmarks together and built the script to automate the tests. We aimed to deploy each framework/platform according to best-practices for a production environment and then stress test common operations: JSON serialization of objects and database connectivity. We were surprised by the wide spectrum of performance we observed and hope that this interesting to you as well. Four orders of magnitude in one of our tests!

If you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.

View on HN

I'm no expert, but I think certain languages/frameworks are better suited to be behind certain servers when high concurrency is tested. E.g. from http://nichol.as/benchmark-of-python-web-servers it seems django would be better served behind gevent.

There are a lot of variables and tweaking that can be done, and it would be nearly impossible to optimize each.

Similarly, I was wondering what sort of an effect connection pooling would have, as the out of the box django distribution doesn't do that. It really didn't perform too well in their tests.

At LEAST gevent with session write-through caching, psycopg2pool, Postgres SQL (hello, excellent South support?), no unnecessary middlewares or applications that rely on them (if it's a speed oriented use of Django, we're hosting on a specific API sub-domain, right?). At most, THEN you tune the settings to have an optimum number of Postgres threads staying alive and tweak some gunicorn/nginx max connections parameters for your site. If running all locally, use UNIX sockets. This article is trash when it comes to providing any useful data other than Django that's barely been configured beyond not using SQL-Lite, and who the hell uses that in production, so I don't buy their argument about "oh well we just wanted to see what It'd do out of the box" rhetoric. Might as well benchmark ./manage.py runserver. I wish they'd it right or don't publish, let alone publish to shill their company that doesn't provide what they advertise.

Thanks for your pull requests, knappador. We will try to get a revised post out in roughly a week or so with as many of the tweaks we've received (as pulls and tips) as we can muster.

I'd be interested to see performance for Vert.x on its other hosts (this is the JVM version, I believe).

Interesting point. You're correct, we've only tested Vert.x as a Java/JVM platform.

I agree, seeing vert.x with it's other language options would be interesting.

Rhino with RingoJS would be another good JVM-based test.

I think you may misunderstand Vert.x's polyglot features:

While vert.x supports many programming languages, all of these are run on the JVM runtime. This means when you use the ruby vert.x API, you're using JRuby; likewise with Javascript run through Rhino, Python through Jython, and Groovy/Scala run through their own interpreter/compilers.

That said, it would definitely be interesting to see the performance implications of using one of those languages and vert.x on the JVM.

Erlang would be nice to see, with, say, Chicago Boss.

Agreed. As you can imagine, we had to stop adding additional frameworks somewhere or we'd never get this posted. :)

Since these tests seem to be all about JSON serialization, it would've been interesting to see the tests with rails-api instead of the standard Rails stack:

https://github.com/rails-api/rails-api

What webserver were you using on JRuby? Was it Trinidad? Did you try Jetpack?

I concur, and Rails 4 may not be officially released yet but it's stable enough to run these tests against.

rails 4 with some concurrency would be interesting to see

Reply on news.ycombinator.com