Rails 3.2 Performance: Another Step Slower
williambharding.com
williambharding.com
The trick with pinpointing a cause is that when I look at the data NR provides, every partial rendered and every action called is slower than it had been previously. Combined with the immensity of our codebase, it's going to be time-consuming to put a finger on a single repro-able cause. But we'll do our best to isolate and report back.
FWIW, have you guys ever considered creating a performance suite test app? With such sparse Google results around Rails performance changes over time, it could be useful to compare the performance of Rails versions on an apples-to-apples basis if we could see how the performance of the test app changed over Rails versions. If we're lucky, it might even catch some of the performance regressions that have happened in the dot releases previously. Even if not, it would at least demonstrate that performance is an important consideration in Rails' evolution. I'd sleep better knowing that.
Maybe take a popular mid-sized framework like Spree and see how it performs on 3.0, 3.1, 3.2, etc.?
I am actively working on getting a long term perf benchmark using the Discourse bench. Server is already provisioned by ninefold. see my talk (towards the end) http://www.youtube.com/watch?v=LWyEWUD-ztQ
Thank you!
What happened is that average response time fell from 415ms to 255ms and across the board everything was faster. This Friday I upgraded ruby on our servers from 1.9.3-p327 to 1.9.3-p448 and that triggered the speedup.
Ruby was installed using rbenv/ruby-build and my guess is that the new version was compiled using compiler optimization flags whereas the old one for some reason was not.
Might be worth investigating :-)
Getting a performance increase during an upgrade would be a highly welcomed change in our world.
I did a writeup on it a bit ago here: https://www.coffeepowered.net/2013/08/02/ruby-prof-for-rails...
Even if we just arbitrarily said "let's focus on whichever method is cumulatively taking most time," we have no reference point for whether Rails could make that method faster or not. The time taken by a method or instantiation is just some number, and whether that number is "good enough" is ultimately a judgement call in which I have little basis for comparison.
I hope I'm wrong and that there are some big obvious things to optimize, but when I've gone down these rabbit holes in the past, my experience has been that there's usually so much data and so many pieces things contributing to slowdown that it's hard to find high-impact things to fix. Especially when it comes to a framework as multilayered & complex as Rails.
I have to wonder what you're doing that's giving you an average of 480ms? Have you stripped out middleware you're not using? Have you looked into exactly where the time is going? There are so many places to start, it's hard to offer much concrete advice.
I think that all the metaprogramming magic that Ruby allows is at odds with speed. That is not to say that it's impossible to make it fast, but it's probably tricky.
I don't think so, Java and C# also have metaprogramming (reflection, class loading, bytecode manipulation) and they are still plenty fast.
Ruby's liability is that it's dynamically typed, and this puts a hard limit on how fast it can be (and also on how toolable it can be, but that's a separate discussion).
If you do a lot of reflection in Java then you'll end up with some pretty crummy performance, but the average Java program doesn't have much.
That said, JavaScript is not far off of Ruby's monkeying capabilities and yet pretty darned fast.
Lisp + Scheme: I mean if nothing else, people have been trying to make them fast for the last more than half of century. Plus AFAIK most Lisp meta-programming code is generated during compile time through macros not during run-time.
Dylan: I don't really know anything about Dylan but it seems to have come out of Apple and CMU, so there's that. Plus it seems to be just a Lisp dialect.
Smalltalk: You consider Smalltalk fast? Lol.
Well, any Smalltalk JIT beat the standard Ruby implementation hands down, what can I say?!
Well, Javascript allows almost as much metaprogramming (and with ES6 proxies and such, exactly as much).
b.) It also has Google pumping a LOT of resources into making it fast.
OTOH, grep for instance_eval and class_eval in rails source.
> OTOH, grep for instance_eval and class_eval in rails source.
I mean what's your point. Yes, I realize that both are used profusely, which is one of the things that this discussion is about.
I didn't describe metaprogramming, just how I usually see common metaprogramming "magic" implemented in javascript. The only reason you'd really need eval in js metaprogramming is if for some semantic reason it was nicer to accept a String of javascript somewhere. Afaik eval doesn't "get" you anything in javascript (it doesn't "unlock" private closures scope or anything, and context-injection is available using #call or #apply) and I almost never see it used, unless I am forgetting something (totally possible, I've been up all night)
> I mean what's your point. Yes, I realize that both are used profusely
My point is I don't see eval used profusely in javascript, which contradicts your assertion in a).
Edit: nvm, I stand corrected: (function(){var x = 1; eval('console.log(x)'); })() prints 1. Neat. But still, I do not see this often used, the callback + context passing approach is much more common.
For a while Phusion offered Ruby Enterprise Edition[1] which made improvements on the Ruby 1.8.7 interpreter including improvements such as:
> * A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
> * The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed. > * The ability to performance tune the garbage collector.
> * The MBARI patch set, for improved garbage collection efficiency.
> * Various analysis and debugging features.
They've since dropped support of it because:
> * A copy-on-write patch has recently been checked into Ruby 2.0.
> * Many of the patches in Ruby Enterprise Edition are simply not necessary in 1.9.
So it sounds like many of the improvements offered in Ruby Enterprise Edition 1.8.7 have been rolled into Ruby 2.0. Do you have any suggestions of what might be included in a Ruby Enterprise Edition 2.0?
[1] http://blog.phusion.nl/category/ruby-enterprise-edition/
There will be no Ruby Enterprise Edition 2.0. REE has been discontinued.
The original goal of REE was to incorporate our copy-on-write friendliness patches into Ruby, and to distribute it in a user-friendly form. Since then, its goal has been extended to include other useful patches as well. We've discontinued Ruby Enterprise Edition for the following reasons:
1. Ruby 2.0 incorporates (or obsoletes) all these improvements.
2. Ruby 1.8 is no longer supported even by its upstream authors.
3. We have limited resources and wanted to focus on Phusion Passenger. Since the discontinuation of REE we've made tremendous improvements in Passenger.
The plan was to hand over maintainership to another interested party. Unfortunately nobody volunteered. http://blog.phusion.nl/2012/02/21/ruby-enterprise-edition-1-...
Even Twitter went public recently and posed to challenge big guys such as Google, Facebook, they've lost my respect as a technology company.
Actually, they escaped to the JVM, and if their job reqs are any indication, they hire massively for Java engineer positions and hardly at all for Scala ones.
I wouldn't call that chickening out, more common sense.
Rails is great for prototypes and toy apps, but once you start needing scale, it's simply not up to the task.
This gets said here a lot. What about GitHub and Shopify, and ...? Are those toy apps?
[1] http://www.slideshare.net/jduff/how-shopify-scales-rails-204...
*From an episode of Ruby Rogues with Zach Holman
You seem to imply Ruby's/Rails' speed matter?
Usually, the DB is the limiting factor in most web applications, few should need to execute more than a few milliseconds of (even) Ruby per page load. That is why people use scripting languages, after all.
Shouldn't this slowdown be a problem in the ORM for this version? I'd look at access patterns for normal requests for the new Rails version and the old. How has the SQL changed?
Sure, Twitter is different since that probably is processor bound.
(Since RoR is a popular framework with lots of eyes, there shouldn't be something simple with e.g. configuring the web server, locking etc.)
I don't accuse them of not saving the world, but Ruby and Rails lost a great opportunity to dominate the web community. If Rails were as performant as Node.js, I doubt lots of companies would change gears.
They didn't lose this opportunity because of Twitter, they lost it because the fact that Ruby is dynamically typed puts a hard limit on how fast it can be. Twitter tried very hard to make it scale before making the decision to switch to Java, and they just couldn't do it.
The task was just impossible, switching to Java was the only reasonable decision given their constraints.
Are you aware that Hotspot, Sun/Oracle's JIT compiler was actually developed for a Smalltalk dialect, Self?
And language benchmarks don't tell the whole story. In Java, idiomatic and typical code will be optimized very well whereas in Javascript such code runs easily at least an order of magnitude slower than its potential. Try to run some idiomatic JS through https://github.com/petkaantonov/nodeperf/ and see it explode :)
So even if Javascript can be shown in some benchmark to close on the JVM, the benchmarks are ignoring the fact that you cannot write Javascript carelessly to get anywhere near those speeds unlike with Java.
Seems that node.js outperforms the JVM for WebDev nowadays.
But they didn't magically make Rails fast, so screw them?
Dynamic webpages consisting of several widgets are quite amenable to parallelization by rendering subviews separately and just assembling them in the layout in a final step. With parallelism slow single-threaded performance can still yield reasonable response-times if you can throw multiple cores at the problem.
But even if you have parallelism, a slow single-threaded GC would halt all of those threads, again affecting your response times.
I don't know about V8, but mozilla's JS is suffering from similar issues. They're currently working on generational GC and it's promising to provide quite some speed-up. But the javascript runtime, just like ruby, was not designed with parallelism in mind, that's why we're seeing those webworkers which essentially spawn an isolated javascript environment. Simply because it's hard to tack on parallelism as an afterthought.
Similarly one of Ruby 2.1's major performance boosts stems from the generational GC.
Charles ominously called "dark matter" inside rails.
Almost all Rails apps that are slow on performance suffer from architectural flaws or oddities in the code. A lot of time Rails get a beating when it boils down to slow, illformed SQL queries. One time I had a customer who wanted me to rewrite their platform because "Rails was so slow", when in fact they did some heavy image processing and uploading to a cloud storage within the browser request. I'm not saying OP does any of this, but since I don't know any details about their codebase or infrastructure I'm making some general remarks that may or may not apply.
Ruby code can be written in so many ways, I guess that is its blessing and its curse. There are bad ways to do things, and there are good ways. With great freedom comes great responsibility.
I would encourage OP to push on and upgrade to Rails 4.0.1. The gap between Rails 3 and Rails 4 isn't that huge at all. I'm sure you'll be pleased.
And you really need to think hard about caching. There are many layers at which you can cache stuff and when you get into caching parts of pages, there is a whole architectural design issue around how to divide things up.
In solving these kinds of problems Rails and its overly simplified ActiveRecord pattern, just don`t give you much wiggle room.
Django is great for consultants/contractors who need to crank out work fairly quickly that may be quite similar to other jobs they've got on the docket. It's great for new programmers or programmers who aren't familiar with the OS-level interactions between a web server and a programming language. It's great for apps with very low complexity (current and projected).
Once you start getting out of these use cases though, IMO, you are asking for trouble getting too invested in the Django (or Rails) ecosystem. It's worth it to learn about how the web server interfaces with the language runtime, so that you can capitalize on microframeworks. When you buy into a framework, you're buying into a huge set of opinions about really important things that have been made based on being flexible enough for about anyone.
I love Django for what it has enabled me to do in my life, and have learned so, so much from getting into its guts to solve problems. That said it would take a lot for me as a salaried, in-house software developer to start a new project using (Django | Rails).
All technology has its trade-offs. Rails can be plenty fast, it just all depends on how you're using it.
My first rails app is approaching 4k lines of code and responses that are cached using fragment caching are rendered in 5-8ms usually on a micro EC2 instance (aka. really bad hardware). I'm using MRI with rails 4.0.1 and I have not done any tweaking. Just basic caching that was trivial to implement and running rails in production mode which is just setting an ENV variable.
But that's not rails' fault, we all know that ruby itself isn't the fastest language to develop in, the "bloat" of rails just increases this a little.
As long as you have enough cheap hardware to throw at all these problems, everything should be fine, though.
I already setup sidekiq to work with e-mails and have whenever being used to generate a new sitemap once a day.
And tbh I don't think I'll ever go too crazy with highly custom analytical queries either because google analytics is quite strong with custom event trackers and tools like new relic are excellent for system health and performance metrics.
As for highly dynamic views that can't be cached then I would worry about them when the time comes. Maybe there's a way to setup varnish or nginx with SSIs to deal with that, it's something I never researched in depth but have to assume is a solved problem at this point with a little elbow grease and technical knowledge.
Without exact profiling information its very hard to fix these issues.
http://www.techempower.com/benchmarks/#section=data-r7&hw=i7...
No surprises here.
I don't know why the Rails community gets their panties in a bunch every time someone brings up the poor performance of rails. It is a a slow bloated framework that requires you throw hardware at your scaling problem instead of writing decent code. In the end, there is only so much hardware you can buy before your app stops scaling and you need a complete rewrite, the way Twitter did.
Ruby is slow, and Rails is slow, although they are both much better than it used to be.
But this isn't bloat. Rails is a large framework that provides moderately sensible defaults. The tradeoff of that is there's a lot of stuff in there which might not be used by every app. This, and the overall flexibility of the framework, can cause performance to suffer, and combined with Ruby's relatively lacklustre performance, that can cause big scaling problems.
Twitter is a poor example though, given the somewhat unique problem space it's tackling. But - building a standard e-commerce or service site? No problem. Getting bigger? Add some hardware. Still not enough? Start customising the framework. Because it's a toolset, not a one-size-fits-all solution. And if you've got outsize scaling requirements, you are going to have to get your hands dirty.
The difference between 3.2.13 and 4.0.0 was +121k/-103k https://github.com/rails/rails/compare/v3.2.12...v4.0.0
So yes, it has grown, but only a little, and may even be a bit smaller, depending on how you compare.
More likely than not you probably have a couple nasty requests that should be handled by a background task and a few more or less caching calls.
Additionally, from your user's perspective, 300~ish ms is already pretty good, I wouldn't worry about that too much. You use Ruby on Rails because it is speed on speed for rolling out features. Focus on a better user experience with pretty good performance and you guys will be alright.
Last time I used NewRelic they gave you an option of sending anonymised statistics to the rails core team.
The stats given in support of this claim are very weak as well. An over-all average is pretty useless for determining the cause. It could be over-all faster with more high outliers skewing the average for some reason. Need more detailed information.
On a side-note, it took me 3 tries before I got the site and not an HTTP 503 error which is rather ironic given the situation.
It this purely a rails upgrade? Or is the OP upgrading other gems at the same time?
You're using newrelic. Use it, and figure out exactly where it's getting slower. Do an entire stack trace. You can figure out exactly where those pages have gotten longer responses. Then you can suggest to the rails community what's slow for your application.
The platform is only as good as the community makes it to be. Maybe it's missing your case's optimizations because you haven't given specific feedback?
Also in New Relic you can look what's slower (ruby, db, rendering, request queuing, ...). Could you give us more details.
Also your average is 423ms, but what's fastest pages like?
Railscasts has a bunch of episodes covering performance related topics http://railscasts.com/?tag_id=1
The real lesson is: if you can write your app in a much faster framework with equivalent development speed, why even waste time to write it in Rails.
Performance is apparently not Rails' only problem.
In 2016 you're gonna love Rails4 - its got some nice support for caching and client side rendering.