Patching Rails Performance
engineering.heroku.com
engineering.heroku.com
using the current libraries, for searching way to improve rails performance, where to you think there are hot places where is easy to go and improve?
second one.
I know that there is plan for rack2, and some clean-up. Can we dream to a more faster rails5?
last one. Not related with your performance improvements.
I found most of the time that AR is quite slow, and I always tend to use ActiveRecord::Base.connection.execute("query here"). What is your opinion about http://sequel.jeremyevans.net/ ?
I tried to write some fancy libs to pull out easy optimizations https://github.com/schneems/let_it_go. It wasn't dramatic but it did find some things to optimize. This lead me to: https://bugs.ruby-lang.org/issues/11375 and https://github.com/rails/rails/pull/20946, though again neither is game changing. I really like this collection of micro optimizations: https://github.com/JuanitoFatas/fast-ruby. I would love to see a tool written that auto suggests code changes. I like http://github.com/schneems/derailed_benchmarks which is what i used for finding object hotspots in that PR. Ultimately I am counting on the VM to make my world easier and faster, as I mostly like writing ruby code, not mangled and optimized barely recognizable ruby code.
> Can we dream to a more faster rails5?
I think we're to the point where there's not many low hanging performance optimizations at the low level. Most apps that are really slow are slow because of their codebase and not Rails, that doesn't mean we shouldn't make Rails as fast as possible. Many of these optimizations I made here were found based on my apps use of the Rails api. I'm hoping others will benchmark their apps and look where we can save a few fractional milliseconds here and there. It's all the small speed-ups that make an overall faster product.
> I found most of the time that AR is quite slow
There are different APIs that are faster than others, i believe the `Model#find` where you pass in a primary key is pretty fast. I actually had a hard time finding any optimizable code in AR, MR. Patterson and Sean Griffin have done a great job of optimizing, though each codebase and use case is different. I've used sequel and really enjoy it. Jeremy writes some amazing code https://github.com/mime-types/ruby-mime-types/commit/8ce1add... and is a great human.
I often wish for a pared down database interaction API (I hesitate to call it an ORM) that just exposes raw data, and provides a convenient query interface (like AR does, but also like raw arel). Maybe this is what Sequel does, and I should be using that, I'm not sure. I was excited about datamapper2 and I've looked at ROM[0], but I'm not sure it's what I'm dreaming of. I could probably cobble it together by using existing database driver gems and arel directly. Unfortunately, as is usually the case when considering venturing off the beaten path of Rails, this sort of thing would give up a lot of the "conventions" advantages of Rails. AR pretty much is Rails.
[0]: http://rom-rb.org/
Sequel's model layer is also faster than ActiveRecord's, because much of the additional functionality that ActiveRecord piles onto all records (dirty tracking, single table inheritance, etc.) are available in Sequel via a plugin system. You can enable the plugins you want to use and not pay the overhead of all the others. You can even enable specific plugins for only the models where you'll actually want to use them.
It also has a lot less magic (no association proxies, unless you enable the plugin for them :)), ridiculously customizable (many more options for associations, custom eager loading logic, and so on), and has an implementation of its Postgres adapter written in C for performance.
Highly, highly recommended.
Edit: Oh, and an issue tracker that is almost always at zero, with a very fast response time. As someone who has contributed a patch to ActiveRecord, I can't tell you how nice that is.
To your edit: Ha, I have a PR against arel that is similarly neglected[0]. Maybe an advantage of using a less popular library is that the maintainers aren't so overwhelmed that things get lost in the shuffle!
On the third point, my sense is that the part of AR
that is often expensive is building full objects,
when you often only need a few of their attributes.
> I’m going to show you how you I did it,
Anyway, this is all good stuff and I don't doubt there's an improvement. A little expansion on the methodology would just make it all the more valuable.
Tangential note: In one of the commit comments there's an exchange about the performance of `any?` and it links to MRI source. However, it links to a line in master, so the link will eventually fall out of date. If you press 'y' in Github it'll change the URL to one with a SHA in it. Or maybe pick the release tag so the link doesn't change as the file updates.
Thanks!
For all the tests I used MRI Ruby 2.2.2. Unless otherwise noted, I almost exclusively use the most recently released MRI Ruby version for testing. There shouldn't be a huge difference in most recent Ruby versions. The only thing that comes to mind is that `String#freeze` existed in older versions but was a method call, and not implemented as a parsing optimization (i.e. it would still allocate a string) however I believe that's been in there for a few years. Maybe the generational GC might have an impact.
> It'd be helpful to know what your GC tuning looks like, too.
I used stock GC settings for all the benchmarks
``` $ env | grep GC # nothing here ```
> A little expansion on the methodology would just make it all the more valuable.
The patch does have some more specifics than the post, especially the commits. I used benchmark/ips in some of them to demonstrate specifics problems. I can do a larger write up on specific methodology, if you think that would be interesting.
> If you press 'y'
Thanks, i've been trying to get in the habit of doing so, that's a good tip.
Good to know. I guess I'm suggesting that specifically calling that out in the text will help it stand the test of time. Otherwise the reader needs to cross-reference the date with the MRI release list.
And really, this is all I was looking for in the methodology. The post covers what you did in pretty good detail. The GC thing was something I was curious about just because that's ping-ponged quite a bit throughout the 2.x releases.
Thanks again!
Thanks for all your work!
Puma is great if you have a Ruby implementation without a GIL (e.g. JRuby, which it was built for), but it's not so good on MRI, because you have to use clustered mode, which runs multiple processes. What they don't tell you is that when running clustered mode, the requests are randomly distributed to each process, and then the requests queued at that level. Each process can run multiple threads each processing a request simultaneously, but because of the GIL anything that's computationally heavy will cause the other requests in that process to run slowly.
That means that it's possible for one process to become backed up with requests, while the other processes on the dyno are sitting idle. And it's actually not just possible, but likely when under reasonable load.
So there is actually no deployment configuration of Puma which makes sense under MRI, at least without a reverse proxy on the server handling queuing:
- Single process (no clusters), 1 thread: Bad because you use only one core, but consistent performance assuming you're not on a multitenant 1X or 2X dyno.
- Clustered, 1 thread per process: Bad because requests are queued at the process level, so a slow request will cause any others sent to that process to have to wait. If you have a request which takes 25s to execute, the one sitting behind it in the queue is highly likely to timeout, and this can cascade to the next request, etc.
- Clustered, multiple threads per process: Better, but still bad, because while multiple requests will be running at the same time in the process, they are competing for the GIL and can slow each other down so much you'll still get timeouts.
The best solution we found to all this was to run a buildpack with a nginx reverse-proxy to a Unicorn server. It's nice because each request is run in it's own process, and the requests are queued at the dyno-level by nginx until any of the processes is free. Combine that with Performance dynos, so you can run a small number of dynos with lots of workers each, where each dyno has consistent performance, and the queuing issues mostly go away.
As you may be able to tell, I've spent far more time debugging this behavior than I would have liked to!
Puma is also great if you have blocking IO like database queries or file uploads which most ruby/rails apps do. The GIL is released when Ruby does IO so it can context switch and run other threads.
> there is actually no deployment configuration of Puma which makes sense under MRI
I disagree, puma makes lots of sense on MRI but it is app specific. By this same argument Sidekiq would be worthless on MRI but it's the dominant work queueing library even though it is solely threaded. In my experience Puma with MRI using "hybrid" with clusters and threads is faster than Unicorn (or rather you can get more throughput and less queueing). However it's app specific. If your app is mostly doing number crunching in Ruby code and not touching IO, then the GIL never gets released and you don't gain much by using threads. Again most Ruby apps use a lot of IO.
> anything that's computationally heavy will cause the other requests in that process to run slowly.
Exactly.
One of the main reasons I pulled recommendation for Unicorn was it's lack of slow client handling, it doesn't help to have a fast webserver, if all your workers are...waiting...on...bytes...and...can't...do...anything...else. Nginx handles slow clients for you so nginx + unicorn is good, it's a harder deployment configuration and it's easier to tell our customers to configure one thing instead of 2, especially if that 1 thing yields better results for the majority of use cases.
I'm glad you got things working with nginx + unicorn. I'm a big fan of loading up PX dynos with lots of workers :) I hope some of the above helps clear up a few points for future readers. I always encourage people to try different options and take our recommendations with a grain of salt. It's also why we've kept our Unicorn article so that anyone not happy with puma performance can try another webserver. Thanks for sharing!
RE threads on MRI, I think it comes down to what your latency requirements are and the type of work as you said. I'm happy with sidekiq on MRI because it doesn't matter hugely if the process gets bogged down for 60 seconds while a ruby-heavy job runs and the other jobs run slowly - jobs are only pulled from redis when there is a spare thread, and sidekiq jobs won't timeout like web requests do. IO-heavy work will get a boost with threads on Puma, but if you get a single Ruby-heavy request running for 20 seconds serializing something, you are going to get timeouts happen.
I guess it does depend on the type of app you are running - Ours is actually quite Ruby-crunching heavy, because it has to serialize some big json responses, which is still incredibly slow in Rails. Apps which don't have occasional Ruby-intensive requests wouldn't have as much trouble with all this :)
run CodeTriage::Application.routes
For a better view of request break down I would recommend checking out https://github.com/MiniProfiler/rack-mini-profiler. You'll not only see where request time is being spent, you'll see how it "stacks up". The flame graphs are pretty useful check out http://www.nateberkopec.com/2015/08/05/rack-mini-profiler-th... for more info.
I would love to read a post written in simple(ish) terms about how V8 handles string allocation and what string performance optimizations helped the most. Would be really neat to see it compared to python/ruby/etc.
It's also worth noting that the majority of the speed here comes from getting rid of hash and array allocations. The bulk of the object count came from string savings but the bulk of the "memory" came from more complex objects.