Is Ruby Fast Yet?
isrubyfastyet.com
isrubyfastyet.com
Right. Your computer can't be trusted to not throttle its performance due to crap cooling, and you're trying to benchmark using it?
It looks like JRuby is just run under the system JRE (which is what?) with no additional options; that's going to be slow for most workloads.
It'd be nice to see a lot more documentation around the runtime configurations and what the Rails app itself is doing.
I just don't really see how this answers the question other than 'ehh I think it looks a bit faster than it was awhile ago?'
EDIT: Also interestingly, JRuby seems to be slightly better on the small instance EC2 test at techempower than regular Ruby (small instance should be similar to a laptop?). Yet in these graphs it's much slower. Anybody know the reason?
Maybe JRuby uses more memory in total, but can handle more concurrent requests per MB of memory used? That could explain the discrepancy if it's true, though it's just speculation on my part.
Another theory is that the benchmarks are too application-specific. Rails is a complex framework with which you can build many different kinds of applications, each with its own performance characteristics. Conceivably, application A could perform better on JRuby, while application B could perform better on MRI. Depending on which application you use for your benchmark, either interpreter could appear faster.
This calls into question the utility of benchmarking an entire framework. It might be better to do micro-benchmarks, with each one testing a specific scenario. Scenarios could include, for example: A request that loads 1,000 database records into memory; a request for a static image; a request that queries a many-to-many relationship; a request that saves an uploaded file to disk.
Testing whole Rails apps really does seem like the best way to get good data. The interesting speedups are likely going to be in places like string matching / string copying and other places that don't involve network access.
Thanks to some community experts, we're now using TorqBox (TorqueBox 3, built on Undertow [1]) for our JRuby tests. The JRuby community seems satisfied [2]. :)
[1] http://torquebox.org/news/2013/12/04/torquebox-next-generati...
Since TorqBox/TorqueBox 3 is very new, I don't suspect it's being used much yet.
[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
Startup time is slow because it needs to warm up the JIT. Same with Rubinius, which is mysteriously missing from most of these graphs.
Memory consumption partially depends on how much you tell the JVM to use - it'll run the garbage collector more or less aggressively to keep within that bound - default appears to be min(system memory/4, 1GB).
That's not to say it doesn't need more memory than MRI, but you should factor in the concurrent native threads allowing one process to serve many clients, and the precise compacting GC limiting memory fragmentation and associated heap bloat with long-running processes.
Question (I'll end up simultaneously googling...): if the code doesn't change between restarts, is there a way one can preserve JIT statistics, so the JVM doesn't have to rediscover hotspots? Like an RDBMS able to load custom statistics for its plan engine.
I think this guy's numbers are very suspect.
I never took the time to figure out why it is faster, but I assume that it is due to C libraries being faster for json serialization and my apps doing primarily json serialization after fetching from the database.
Edit: I should test the performance of the json libs, which would be easy to do
That said, we really need to know which flags he's running Jruby with - and which
The internet tells me that you achieve better performance with -server at the expense of a bigger start-up time penalty. Anecdotally, it takes a while for the JIT compiler to find the hotspots in your code and optimize them away.
All of this will depend on how Jruby-rails keeps code in memory, yadda yadda. Someone who actually has used Java/Jruby recently can chime in with more details.
As far as I know, Passenger Enterprise and Puma are the only multi-platform ruby rack app servers that support multi-threaded concurrent request dispatch. There may be other jruby/java-only deployment scenarios that support concurrent request dispatch.
Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!
That's a good point.
That said, JRuby would most likely benefit from a concurrent test, as the thread-per-request model of JRuby can scale much higher than the process-per-request model which is used a lot in the MRI world.
Of course, it's not a real Rails app presumably, but just a sample/dummy one. So you may or may not see comparable speed-up with a real app. But it's probably suggestive of the upper limit of improvement you can get by switching ruby versions.
"Will I get a performance improvement from my Rails app by upgrading ruby" is a real question a lot of people will have. Micro-benchmarks of ruby wouldn't answer it as well as the OP does.
Kudos to making the benchmark opensource at least, but even from a quick peak, I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be this slow. Odds are the benchmark is throwing away the JIT gains or the configuration is poor. Add that to the fact that there is no discussion of what a 'rails request' is or what version of Java is being used and there seems to be little to learn from this research.
But I agree it is hard to get benchmarks right -- I'm not even sure there is a 'right', as a benchmark will usually be based on certain assumptions that only sometimes apply. (Unless you are benchmarking your own actual production code, in which case you know it's assumptions are valid -- for your own code).
So I don't think that's a reason not to share benchmarks. But I _do_ think it's a reason to be really transparent about exactly how the benchmarks were done. Ideally all code should be available on github.
Then people can examine your assumptions (including ones you knew you were making and ones you didn't) and modelling, as well as fork or pull request to do benchmarks in some other way to better model the scenarios they care about.
Although also in my experience, you get LOTS of people telling you your benchmark is wrong because of X, Y or Z -- sometimes they are correct that it doesn't accurately capture something they are interested in, sometimes they are just confused -- but virtually nobody actually interested in writing code via fork or pull request to capture what they are interested in. Once you put out a benchmark, everyone in the peanut gallery thinks they are entitled to keep demanding you change it in various ways for them.)
No matter how much time you spend on it and thought you put into it, publishing a benchmark on HN is probably the easiest way to have lots of people telling you you're an idiot who did things wrong. It almost seems like HN collectively wants to discourage people from running benchmarks -- or from sharing what they found publicly.
But it's because doing good benchmarking is so hard -- well, that's exactly we need more people working on it, and sharing how they did it so others can improve upon it, etc, right?
I happen to have Ruby 2.0 installed, and I was surprised to see it handily beat the other languages. I had expected Ruby to be the slowest of the pack, but they've apparently made big strides in 2.0.
On the off chance that anyone is curious, here's the results of running them on my machine:
binary_trees - wren 3531 0.28s
binary_trees - lua 1366 0.73s 258.40%
binary_trees - python 1420 0.70s 248.64%
binary_trees - ruby 3170 0.32s 111.39%
fib - wren 2688 0.37s
fib - lua 2998 0.33s 89.65%
fib - python 1390 0.72s 193.38%
fib - ruby 3645 0.27s 73.74%
for - wren 9092 0.11s
for - lua 9387 0.11s 96.86%
for - python 3171 0.32s 286.74%
for - ruby 8319 0.12s 109.30%
method_call - wren 4703 0.21s
method_call - lua 1780 0.56s 264.20%
method_call - python 806 1.24s 583.33%
method_call - ruby 3408 0.29s 138.02%
[1] https://github.com/munificent/wren [2] https://github.com/munificent/wren/tree/master/benchmark
https://github.com/JuliaLang/julia/tree/master/test/perf/mic...
I had JS (using node) in there for a while, but it was regularly 10x compared to the other languages. Using a JIT makes it effectively a different class of implementation. It's like entering a car in a bike race.
Thanks for the benchmarks! I might steal some of those. They may be more arithmetic focused than I need (which makes sense for Julia, but less sense for an embedding scripting language where most math is probably done in the host app).
I had actually thought that's what I was doing when I compared it to Lua! I was quite surprised at how well Wren did in comparison. The trick will be to see if I can keep it that fast as the implementation gets more robust. It takes some shortcuts right now that don't scale well, but happen to be faster.
> I really want Jeff to write a little book about it, walking the reader through the implementation.
I would love that. I've skimmed femtolisp a few times and it's clean but suffers from the almost total lack of documentation that other programming language implementations also seem to have.
Programming language implementors do seem to be even less fond of writing comments than most programmers, don't they? I think one of the issues is that internals can change a lot fairly quickly and dealing with comments just makes that worse.
Lua 5.1 initial release was in 2006[1]
Python 2.7 initial release was in 2010[2]
Ruby 2.0 initial release was in 2013[3]
[1] http://www.lua.org/versions.html#5.1
[2] http://en.wikipedia.org/wiki/History_of_Python#Version_relea...
[3] https://www.ruby-lang.org/en/news/2013/02/24/ruby-2-0-0-p0-i...
Wouldn't it make more sense to do this on generic server hardware with a LTS Linux OS? Once that vintage and customized MacBook bites the dust the website pretty much dies as well.
As a small example, I benchmarked a real Scala web app running on worse hardware (a 1GB Linux VM) at over 650 requests/s. There was no degradation in load at this rate of requests, but it was 10x the load we saw in production so I stopped there. The benchmark of Rails here goes to ~120 requests/s.
see: http://samsaffron.com/archive/2013/12/11/call-to-action-long... and http://community.miniprofiler.com/t/is-ruby-fast-yet/197
"x is better" where x is more requests, faster startup time, or less memory usage.
I prefer graphs that lead me to what I should expect and that sentence would go a long way to eliminate the "mental math" I had to do to understand each one.
On another note, the 2 year graphs are particularly interesting. They're more consistent than not but those inconsistencies seem a little awkward.
Also the benchmark is the best case. In real world projects, I've seen Rails take over 60 seconds to start (it took a few days of development effort to bring it down to ~20 seconds). Every time you rerun tests, run a rake task, start or restart your server, etc. It's like the heartbeat of the development process grinds to a halt when you add more features to your Rails server.
And no, tools like Spork are not the answer, since you still have to balance the amount that is resident in memory to fork with coverage of what you can change. It also opens up a fractal of complexity.
I'm assuming the negative votes are due to the formatting and not the message. I hope you, dear reader, don't miss the irony of this situation...
If it's downvotes because of the message, well I feel bad for the downvoters and wish them much luck with using their framework.
It's not exactly a moral imperative to obsess over your HN comments, IMO.
Most likely, it's a simple bug, that seems to be fixed now :-) No ethics involved. Nothing to see here.
Btw, in the OP, I was trying to express the effect that the increased startup time affects flow and iteration. It's a similar effect to a comment breaking the layout of the page.
I just learned something new today :-)
Also, I'm a bit surprised that this doesn't include more implementations of Ruby. Where's Rubinius? IronRuby? REE? I hope the list gets updated.
Furthermore, the benchmark itself could be better; I won't say I am disappointed, but I am not too thrilled either.
I'm building http://techendo.co/ and it's running rails on heroku. I was wondering if there was some easy ways to get some better performance out of it immediately now that we're picking up traffic. If anyone could give me some pointers or link me to a page I'd appreciate it.
* More instances will give you more requests/second, but only so long as secondary system (database, search server, whatever) can keep up with the load
* N+1 queries will kill your performance. They can add a lot of time to processing requests.
* Everything in the most used parts of your site should hit an index on the database. If it's not I can almost guarantee it will be slow.
* Look for work that you can move "offline". As long as they go out within a minute nobody will notice that you aren't sending emails at the exact time they are being generated. Move that work to a background queue so your web server can just add a queue entry and get on to the next request.
* Cache anything you reasonably can. It's a lot cheaper (i.e. faster) to read from a cache than to do a database lookup. It's cheaper to read from a cache than to render an ERB template. Caching can introduce its own difficulties, but the performance gains can be worth it.
Without knowing some specifics of your implementation I really can't give you better advice. Depending on the specific libraries you use it might not even be possible to move to JRuby due to support for C extensions. You might get some performance benefits out of it, but I'd expect it to only buy you some time to execute on some of the above improvements.
Relevant RailsCast: http://railscasts.com/episodes/368-miniprofiler?view=asciica...
You can read up here for more:
https://devcenter.heroku.com/articles/moving-an-existing-rai...
http://stackoverflow.com/questions/151595/jruby-on-rails-vs-...
So it's not just a 5 second thing, but if you have a bit of time and need performance it's something you can try.
Fixing bad queries (n+1 is a common thing that will get you in trouble), and caching API calls (Rail.cache.fetch is your friend) will help a ton.
Background any tasks that must be slow. I like using Resque/Redis myself.
At some point it helps to cache HTTP requests when possible too. I've been working trying to get a special Buildpack working with Nginx for better caching on Heroku, but I'm not done yet. Also make sure that you aren't serving static assets from Heroku, and you're using something like CloudFront for it.
Email me if you need more help. I can normally make these things fly.
And the answer is no, and unlikely to be any time soon:
http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...
That said, +1 for having a go at it!
Also, yes, Ruby is pretty slow. But it's fast for getting something up and running quickly. If you get wildly successful, you can always swap it out for something else if you need to.
It should be possible (if you care enough) to re-run this from the source control history on platforms of your choosing, across whatever date ranges you wish. Anyone really, really care enough?