Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0 | Hacker News Reader

Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0

gist.github.com

27 pointsresca7926 comments

This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.

Also, this is in dev mode... did you disable Rack::Lock?

MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)

Rails 4 is thread safe and I used puma as server Also the app is in production mode and the db pool is 16

Can you post a link to the app? Have you considered running the TechEmpower benchmark app? https://github.com/TechEmpower/FrameworkBenchmarks

Rails 4.0.0 application starting in development on http://0.0.0.0:3000

This most likely means Rack::Lock is used in your test.

I used the following command rails s puma -e production

Rack::Lock was removed as a default in Rails 4.

> This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.

This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.

Honestly I declare no winner. This is a very simple test. I wanted to share with you, to even get tips thanks

It's a Rails 4 app. Rack::Lock is removed by default.

Reliable benchmarking is hard. There are actual journals about measuring computer system performance.

Some questions:

* How is startup time controlled for?

* Why only one run?

* Why so few requests?

* Why no warmup period?

* Why this particular configuration?

* What is the OS, hardware configuration?

* Why is it running in a development environment instead of a test or prod environment?

Prod. The app is a very simple rails4 standard app

iMac Intel Core 2 Duo 3,06 GHz cpu 1

    => Rails 4.0.0 application starting in development on http://0.0.0.0:3000

.... update the gits .. thanks!

I would really like to see some articles about proper benchmarking procedures at 1000+ requests per second. Are there any?

The ACM's SIGMETRICS[1] are the main group publishing this kind of research. Some of it gets very specialised -- just picking a title from the most recent Performance Evaluation Review[2], I get "YOUQMON: a system for on-line monitoring of YouTube QoE in operational 3G networks".

However, being the ACM, it's basically inaccessible unless you sell a kidney.

An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money.

[1] http://www.sigmetrics.org/

[2] http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT...

[3] http://www.cmg.org/

Sorry to go off-topic, but does anyone know if it's possible to get your hands on these pubs/journals for free at the Science, Industry and Business Library [1] in NYC?

[1] http://www.nypl.org/locations/sibl

I would also love to see an application in production measured with NewRelic or something else and compare the results. A good way to test this would be to spin up three servers.

* 1 load balancer that plays round robin on the requests.

* 1 server running MRI

* 1 server running Rubinius

Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same.

This would give me and others better insight as to how the two interpreters are running in a production.

Although you may be running a single-threaded test intentionally, I generally recommend using a multi-threaded load simulation tool versus Apache's ab tool [1]. My preference is wrk [2], especially since it provides for time-limited tests rather than request-limited. Another alternative is WeigHTTP [3], which is basically a multi-threaded replacement for ab.

[1] https://news.ycombinator.com/item?id=6211883

[2] https://github.com/wg/wrk

[3] https://github.com/lighttpd/weighttp

So, MRI is way faster than Rubinius in this case? Interesting...

yes... and I repeat it and sometimes in this case is 2x faster

Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.

That was my concern - Rubunius' selling point is concurrency, so I'm sure there is an approach to implementation, which is why I've never used it.

No. Rubinius' JIT has not had time to warm up. These tests are a little unreliable. I can't find the article off hand, but the way these tests were done aren't exactly telling me anything. Power of ten tests don't do much other than show me that the tester is naive.

Gist title is Ruby 2.0 MRI vs Rubinius 2.0 While HN Title is Ruby 2.1

So which is correct?

sorry.. I have updated the test with ruby-head Ruby 2.1 MRI 2.1 Rubinius 2.0

very interesting, thanks!

Reply on news.ycombinator.com