A benchmark comparison of Node.js, Tornado, and 3 Erlang servers | Hacker News Reader

A benchmark comparison of Node.js, Tornado, and 3 Erlang servers

ostinelli.net

95 pointsEvanMiller33 comments

This should just fall under the 'duh' category. Quite simply, Erlang is the gold standard for scalability (horizontal and vertical) and nearly everything else is far #2.

However, choosing a tech is much more than just raw numbers. There are so many factors at play and it shouldn't be taken lightly.

That being said, we did something similar about 2 years ago and Erlang/Riak came out so far ahead of the competition for our needs and it wasn't even close.

I did go back to my notes just recently and looked again with Node in mind. Though I didn't do the same bake off we did back then, I did a thorough exercise with Node to make an evaluation and, while a cool tech, would not have made the cut. Erlang/OTP w/ Riak is not hard to program (anyone can learn), takes so much of the guess work out of scalability and is easy to debug. Node isn't quite mature enough to handle this yet and coupled with the true lack of multi-core, can't quite hang with the big boys just yet.

Erlang/OTP is clearly in a league of its own.

Gold standard for scalability? There are certainly others. Check out numbers for Haskell:

http://snapframework.com/blog/2010/11/17/snap-0.3-benchmarks

http://www.yesodweb.com/blog/preliminary-warp-cross-language...

Would be interesting to repeat the author's experiment exactly, but in any case I would expect an order of magnitude upwards difference in performance.

Haskell gives you non-blocking IO without the hassle of callbacks, multithreaded operation and more.

I wouldnt expect an order of magnitude better on the same benchmark actually, especially as the benchmark in question is for a single core. There is clearly some scope to improve over these, but I doubt it is a factor of 10. Run the benchmarks rather than making claims with no backup.

The author emphasizes the first picture (which shows Misultin delivering 3X the throughput of a Node.js server), but I think the most interesting picture is the Response Times (picture #3). Look at the purple and yellow lines. When Node.js starts overloading at 3000 reqs/sec, the average response time drops to ~300 ms (just guessing, I'm not good at reading log graphs). But when Mochiweb starts overloading at 5000 reqs/sec, the average response time is around ~50ms.

In other words, the slowest of the Erlang servers has twice the throughput of Node.js, and handles server overload much more gracefully (at least in this contrived example).

I'm not sure this is surprising.

People use Python and JS-based webservers not for their performance, but because they like the languages. And those languages were designed not for speed or concurrency but for other things, like expresiveness.

Erlang was designed with performance and concurrency in mind, and if it didn't beat the pants off of Python and JS servers, I'd be shocked.

Not that that means it's the right choice for all things. Performance isn't everything, being able to write code in Python is a good enough reason to use something like Django.

Erlang was not designed with performance in mind. It's only gotten reasonably performant in it's later life.

Indeed. I'd really like to see the Haskell servers, warp and snap, thrown in, since the infrastructure there was actually built for raw performance.

I have put together the code to implement the benchmark using Snap and would be happy to share with anyone willing to boot up the instance and run.

I might do so myself if I find the time tomorrow.

That's not quite fair. Erlang was designed from the start to be very, very fast at process switching and message passing. True, that design took a while to bear fruit, but my understanding (from listening to Joe Armstrong speak about it) is that being able to make the core of the language highly efficient was a goal from the start.

Trying to make it as efficient as possible is different from the purpose of the language being efficiency.

Indeed, but in this case both apply.

EDIT: Give this a listen, the early design and development is described in more detail than I'm capable of going into: http://www.se-radio.net/2008/03/episode-89-joe-armstrong-on-...

I agree. It's interesting to see node fans get huffy when Erlang people say, "hey, we're still #1"

But for people who already love JS, node is a dream come true. It's really fast. Not as fast as Erlang, but more than fast enough to justify building massive projects in it. Assuming you are one of those that loves, or believes in, JavaScript...

Don't you find using callbacks for everything to be a straightjacket?

There are some nice libraries that ease the pain of callbacks to a certain degree.

Async[1] (the waterfall method) and Step[2] come to mind.

[1]: https://github.com/caolan/async

[2]: https://github.com/creationix/step

This pretty much spoils the whole idea that you'll use your clientside JS on the server though, huh?

The callback issue means that client and server are really not running the same language. They sorta look the same I guess.

I'm not sure about step, but caolan's async is perfectly usable in the browser, as well as in node.

That said, code sharing between browser and server often extends to templating, validation and a few utilities, in my experience. More complex logic is better served with message passing or something like dnode.

Personally, yes. I like to use eventlet in Python partially because of that. Interacting with the scheduler is implicit in eventlet (and gevent, which some prefer) so you write code that looks synchronous.

On top of that, you get nonblocking I/O almost for free, similar to what node attempts to provide, where instead of real nonblocking I/O the system will make blocking I/O look like it's nonblocking. Scala does this too.

But, users of Twisted could also use inLineCallbacks for a similar feel, though it's not required. I'm curious if Node.js could just build the same?

I found Steve Vinoski's comment on this exercise interesting:

http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-b...

Comment I left on reddit, with my benchmark numbers for the tornado server under PyPy: http://www.reddit.com/r/programming/comments/h7cs9/cowboy_mi...

By using couch-DB you can have Erland I/O scalability and then use whatever you want for the rest of the server side logic.

If you have such a massive success that you need the rest of it also ported, then you can make this investment later on. The rest of us simply don't have the time to learn whatever is the fastest each month.

The fact that CouchDB is written in Erlang is only part of the equation.

You're not considering that CouchDB views are written in Javascript or that CouchDB uses HTTP as its connection protocol, which has more overhead than the typical binary protocol of traditional DBMSes.

Also, avg. response time in CouchDB is directly proportional to the number of view invalidations that occur. If your data is changing a lot your views will be invalidated frequently and for those requests avg response time will drop significantly. Throw in a reduce function and re-indexing can become an order of magnitude (or more) slower.

The author claims that he doesn't want to test an overly-simple "hello world" benchmark, and then he does exactly that.

The author says what he means by an overly-simple "hello world" is a test of retrieval of static resources. His test is very simple but it does require a dynamic response from the server.

Is it me or is this not a really fair comparison since the erlang frameworks would be utilizing both cpu's while node and tornado wouldn't be?

In real life a load balancer in front of tornado or node would be a far better solution than going down the erlang road in production.. support and maintenance wise + more programmers out there knowing the language would win hands down IMO...

If you read the article closely you will notice that the Erlang VMs were started without SMP support enabled; everything was single-core. If you turned on Erlang SMP support then the comparison would be even more lop-sided than the example presented in the OP. In real life a hardware load balancer in front of a pool of Erlang servers would be even better :)

Would be interesting to add AppServer-in-PHP: https://github.com/indeyets/appserver-in-php

It performed quite equally to Node.js on my small-scale tests (http://bergie.iki.fi/blog/php_can_perform_better_than_node-j...)

I would be interested on how the Haskell frameworks like Warp or Snap stand up to Erlang.

Yesod/Warp seems to be incredibly fast according to a benchmark conducted by the Yesod devs. [1] I would also like to see a comparison with Erlang.

[1] - http://www.yesodweb.com/blog/preliminary-warp-cross-language...

What sort of datastore would be a good pairing for this? Seeing as it doesn't matter how fast a server is if the datastore can't keep up. Redis? Riak? CouchDB?

Two comments.

These microbenchmarks don't really do it for me. It would be much more interesting to see a real test application being benchmarked. Something with users, objects, pages that have content.

It would be more fair to look at the code, complexity and performance drops when you actually build something more real.

Also, I know Erlang is awesome and super fast. But personally I would rather spend the money on a good horizontally scalable architecture and extra hardware if that means we can build apps in a more mainstream language like for example Python or Ruby.

Breaking news: Author of web framework writes benchmark to prove his web framework is the fastest.

All three Erlang frameworks beat the pants off node.js. Not just his.

I should have clarified to say that the Erlang frameworks beat both Node and Tornado thoroughly.

Reply on news.ycombinator.com