Hacker News Reader

Comment by xal | Hacker News Reader

What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment?

Update:

Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests.

Update2:

Ok it seems to run in production mode but still, passenger is not an idiomatic choice.

View on HN

For the sake of curiosity, I happen to have recently done a benchmark of a "hello world" rack app (literally just responds "hello world" to every request) on a number of Ruby servers (mostly JRuby, but also Puma on MRI).

They were all run in production mode with logging disabled, etc.

http://polycrystal.org/~pat/scratch/microbenchmark.png

Note that a difference from 10k requests per second seems huge compared to 3k, but if you invert it, you get 100 and 333 micro seconds per request, respectively. In a real, non-"hello world" app, these differences are going to be negligible.

Though perhaps it would be more interesting if instead of just responding "hello, world", the app parsed some query parameters or something. But I was mostly interested in the overhead of different JRuby servers, not comparing different app servers (i.e. overhead from Sinatra should be more or less identical whether you're on Puma, Trinidad, or whatever).

Sheesh, you can clearly see the GC pauses in the Java versions.

We used Phusion Passenger, although we have plans to add additional servers (such as Unicorn). We tried to spend time with various server choices for all platforms, and for ruby, in our short test, Passenger won out against the others.

Our understanding is that when running Passenger, simply passing '-e production' to the command line is sufficient to run in production, but if that's incorrect, we'll gladly update the test.

Please make sure you're setting higher GC limits for the Ruby tests. Ruby's defaults are awful for a framework, and result in a LOT of GC thrash. It's not uncommon to see an order of magnitude improvement in performance when they're tuned properly. (edit: I'll just send a pull request, I found the setup file!)

Something else you might consider is the OJ gem rather than just the stock Ruby json gem. The latter is notoriously slow and memory-hungry (which will compound the GC issues!)

> make sure you're setting higher GC limits for the Ruby tests

Could you elaborate on this, or point me in the right direction? I'm learning Rails and curious.

For anyone still lurking, this user replied to me via email:

Ruby allocates heaps for its objects, and sets GC thresholds based on those heap sizes. Ruby allows you to change those settings via environment variables, which means that you can end up doing fewer allocations and less aggressive GC, which makes sense when using a full framework like Rails, which is going to allocate a lot of objects.

There's a more complete answer here to get you started: http://stackoverflow.com/questions/13387664/ruby-gc-executio...

Pretty sure passenger+nginx configuration is the most common rails deployment. Not the fastest but most common much like apache is the most common web server for php.

If that's true, it's not really a fair comparison. Running in development mode slows things considerably. They do have a question in the FAQ about that point:
"You configured framework x incorrectly, and that explains the numbers you're seeing." Whoops! Please let us know how we can fix it, or submit a Github pull request, so we can get it right."
Perhaps you/we should submit a pull request?

Passenger wouldn't be my choice personally, but I don't think there's anything non "idiomatic" about it. Engine Yard, Cloud 66, etc. use passenger in their PaaS configs, it's been very popular (on the wane now, but still), etc. It seems fair enough and the differences aren't going to be the sort of order of magnitude change which would really matter on this sort of thing regardless.

This (from [1]) looks like passenger in production mode to me.
> rvm ruby-2.0.0-p0 do bundle exec passenger start -p 8080 -d -e production --pid-file=$HOME/FrameworkBenchmarks/rails/rails.pid --nginx-version=1.2.7 --max-pool-size=24
https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Are you sure? This looks like the right file to me, and it says '-e production', last changed 6 days ago: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... (unless that's an incorrect switch / something else overrides it - I haven't tried it, not too familiar with Passenger)

I'm not familiar with Rails. Could you explain what you mean in layman terms? Why is code being reloaded on every request by default?

The Rails framework is written to reload (reevaluate) most application code between requests in development mode, so that changes during active dev are reflected. This is not the (default) behavior in production.

In development mode, it's nice to be able edit ruby code, then hit reload in the browser to see it in action without restarting the whole app. Sure there are other ways to accomplish this, but reloading code is typically how it's done in Ruby.

In development mode Rails will reload code on each request to pick up on changes you have made to your application. That way you can interact with the application to help verify that your code is working properly.
As of Rails 3.2 development mode watches for file changes and attempts to only reload those files, but it's still a significant performance issue.
By default all Rails applications start in development mode, so one gotcha of benchmarking Rails is that some people will forget to set the mode correctly. That said, from the setup code[1] (line 14) it looks like they were running passenger in production mode. The max pool size seems excessive, especially when running on large ec2 instances, but I'm not fully convinced that it's out of line.
[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Reply on news.ycombinator.com