Snap: A Haskell Web Framework: Benchmarks | Hacker News Reader

Snap: A Haskell Web Framework: Benchmarks

snapframework.com

44 pointsaliasaria17 comments

Let me state once again that the Rails benchmarks are absolutely way off. Snap would still be faster - much faster - but you can expect several thousands req/s from that Rails app on that kind of hardware.

It was said in another thread that these were done using development servers: http://news.ycombinator.com/item?id=1371048

Edit: Oh he responded to you the last time, but I'll leave this here so others can see

I did my own testing, but I didn't publish them because I left Haskell completely unoptimized and didn't really have the time to try to optimize it. I also added play (http://www.playframework.org) into the mix, because I admire the framework and how closely it mimics Rails, but in Java (and Scala).

Still, I saved 'em, so here you go:

On a 2.27Ghz Xeon dual quad core with 12 GB ram, using Snap's tests from GitHub:

  Test Pong:
  | Rails Webrick | Rails Mongrel | Rails Thin  | Rails JRuby (glassfish) | Snap          | Play!         |        
  | 155.0 req/s   | 1940.5 req/s  | 619.0 req/s | 3606.0 req/s            | 11347.0 req/s | 19602.2 req/s |

  Test File:
  | Rails Webrick | Rails Mongrel | Rails Thin  | Rails JRuby (glassfish) | Snap          | Play!         |        
  | Didn't test   | 188.3 req/s   | 718.0 req/s | 6633.3 req/s            | memory leak   | 606.6 conn/s  |

The full results are here:

Pong: http://gist.github.com/414724 File: http://gist.github.com/414728

Note, Snap had a mem leak when performing the file test that caused a machine with 12 GB to swap, but that may be because I used the Ubuntu Lucid version of Haskell.

Also note, Play is damned fast. And relative to other Rails options, JRuby with glassfish is as well.

Very interesting - thanks.

Scala is a hot language right now, and when Play gets really solid Scala support (does it have it already?) then Play may get to be more popular.

It supports Scala well, but the API (on the Scala side) is a bit of a moving target.

However, I really encourage all of you to give the Java side of play a shot. It's really unlike any Java framework I've ever worked with, and I've been doing Java for far too long. It automatically reloads your changes without a compile...it makes great use of annotations, and it uses Groovy for the templating language, it comes with support by default for the big three IDEs (Netbeans, Eclipse, IntelliJ) although you can easily be happy with vim, emacs, or tm. After a bit of time, it's easy to forget you're actually working with Java.

Also, if you know Rails, you essentially know Play. The concepts are the same, the directory structure is the same. You're learning the Java version of Rails, so there are differences, but my years of Rails development experience made picking up play a breeze.

edit: clarified API comment

You need to look at the average reply rate instead of the request rate. Requests/sec can be misleading. The replies / sec tell quite a different story. Here they are for pong:

Rails Webrick: 153.9 Rails Thin: 33.3 Rails Mongrel: 90.7 Rails JRuby (glassfish): 3,350 JRuby Mongrel: 31.3 Snap: 11,309 Play: 19,796

You're absolutely correct. Sorry. Here it is, updated properly:

  Test Pong:
  | Rails Webrick | Rails Mongrel | Rails Thin  | Rails JRuby (glassfish) | Snap          | Play!         |        
  | 153.9         | 90.7          | 33.3        | 3350.5                  | 11309.5       | 19796.3       |

  Test File:
  | Rails Webrick | Rails Mongrel | Rails Thin  | Rails JRuby (glassfish) | Snap          | Play!         |        
  | Didn't test   | 179.9         | 758.0       | 6146.9                  | memory leak   | 1617.9        |

Didn't realize Glassfish was that fast. Thanks.

And hadn't heard of Play! Will check into that.

Also, just as a counterpoint to all this - as PG pointed out in his book, with Moore's law, speed of execution is no longer the limiting factor in most software development. It is programmer time and productivity, where (I'm guessing) Rails still wins this test. I'd expect that to be an ongoing trend. But of course speed of execution is still necessary in special situations, so this is still valuable.

I think Play+Scala should be competitive with Rails in expressiveness, while still being a lot faster. Of course, if you prefer dynamic typing you might disagree with me here.

Personally, I've found it's very expressive and easy to follow, although I'm using Java play. And having reliable autocomplete and ctrl+click navigation is something I really missed in Rails, but have in Play.

Surprising that Webrick is faster than Thin and Mongrel. How can this be?

If you can, try nginx + unicorn or passenger. It should be faster than JRuby on Rails with Glassfish.

interesting. i would like to see a comparison to yaws or mochiweb (perhaps w/nitrogen).

Although these seem like useful (and impressive) baseline benchmarks, won't the really important benchmarks come from measuring actual dynamic content generation, and/or caching performance for generated dynamic content?

When I did experiments with RoR versus Haskell via FCGI, Haskell had around a two orders of magnitude performance advantage under load and only required a single OS process.

Would love to see a comparison to Node.js on here, as well.

Indeed, for example the express framework appears to be optimized for high performance http://expressjs.com/

Reply on news.ycombinator.com