Rails 3.2 RC1: Faster dev mode & routing, explain queries, tagged logger, store | Hacker News Reader

Rails 3.2 RC1: Faster dev mode & routing, explain queries, tagged logger, store

weblog.rubyonrails.org

185 pointsaaronbrethorst50 comments

I've really noticed quite a bit of speed improvement lately with Ruby 1.9.3 and now Rails 3.2.0-rc1, at least in my development environment. It's a good thing too because lately I have been thinking back fondly of the good old Rails 2.x days when you could restart the development environment without flinching. With the latest incarnation of these two tools I love the Rake command again. :)

Now, if we could just get someone to commit the fix for the debugger under Ruby 1.9.3. Not being able to debug, or at least having to work too hard at it, is really a drag on my productivity and is keeping me from moving to the newer faster stuff everywhere. Come on Mr. Moseley, release ruby-debug-base19 v0.11.26 already!

I can only hope that running my app in dev mode in 3.2 won't bleed memory 40 MB per request. I have no such issues in production environment and it makes testing/debugging a royal pain.

It's probably not Rails itself...

I had a similar issue, which turned out to be a bad Rails plugin, which maintained a few unnecessary pointers to my model classes. The small plugin caused a 20+MB per request leak in development. I rewrote the piece of functionality I needed from the offending module & the issue went away entirely.

Sadly, there isn't really decent memory profiler support to help you track it down. I found the issue by running `git bisect run` with a little script which restarted the server, beat on it a bit with `ab` and then printed memory usage with `ps`. Took me about 10 minutes to track it down to the exact offending commit.

How did you deal with the migrations problem? I usually can't bisect Rails code because there's no way to migrate from here to there anymore, let alone do so automatically..

I store the schema as SQL instead of the .rb file nightmare. Here's the script to do it quickly with Postgres & bypass the Rake slowness:

    #!/bin/bash

    root="/path/to/your/code"
    user='postgres'
    db='dev'
    schema="$root/db/schema.sql"

    pg_dump -U $user $db \
      --schema=public \
      --schema-only \
      --no-owner \
      --no-privileges \
      --file $schema

    pg_dump -U $user $db \
      --table public.schema_migrations \
      --no-owner \
      --data-only \
      --insert \
      >> $schema

Then a trivial `dropdb`, `createdb`, `psql < db/schema.sql`

And then a small script with 4 or 5 curl calls can get enough data into the database to drive some tests.

Gah, of course. Brilliant, especially skipping rake. (Though, pro tip: With Ruby 1.9.3, set export RUBY_GC_MALLOC_LIMIT=59000000 and you'll save 5-6 seconds at launch.)

does this fix the abominably slow development with the asset pipeline (without the gems needed to hack it and speed it up)?

Basically, can someone give me a way back into pre-3.1 asset pipeline slowdown ?

Can you be more specific? Are you talking about rails startup time or SASS compilation? Or something else?

in dev mode, the asset pipeline induces extremely slow page loads in development (2-4 seconds per page locally)

This has been the single biggest complaint against Rails 3.1 if you google for it. For e.g. take a look at http://stackoverflow.com/questions/8084006/serve-assets-dire...

Similar questions have been addressed by either installing rails-dev-tweaks or by pre-compiling assets - which is then a problem if you're actually working on the UI/JS - which is where half the dev time is usually spent.

Ah, I was wondering how come my development environment had slowed down so much in the past couple of releases. Had I known enough to check for it I probably would have found out about that rails-dev-tweaks Gem sooner.

Anyway, yes I think that this release fixes those issues. However, there appears to be a problem with the SASS @import command and partials[1] so I haven't been able to fully test my system yet with all it's assets working properly.

[1] https://github.com/rails/rails/issues/4041

I've noticed that using Passenger Standalone is faster than webrick / mongrel in dev mode using Asset Pipeline. Give it a shot.

I just switched to pow to avoid the annoying content-length warnings in 1.9.3 webrick, and it's pretty darn zippy, debug-mode asset pipeline and all:

curl get.pow.cx | VERSION=0.4.0-pre sh

Sure; no, I've seen the same kinds of problems.

If you are interested, I posted in the comments to this gist (https://gist.github.com/1184843#gistcomment-65875) some changes that we made to our SASS files to markedly increase the compile time.

For us it went from (depending on the machine) something like 40s down to a <5 seconds which was a massive improvement (CSS development was almost impossible for a while there).

There have been several issues with Sass (sass-rails) and the Asset Pipeline. Some of them involving Compass (https://github.com/chriseppstein/compass/issues/516#issuecom...), some of them due to Sprockets (https://github.com/sstephenson/sprockets/pull/268#issuecomme...).

They have indeed been crippling for front-end development, and majorly under-reported since this summer. So if you're facing any of these issues, please participate in the GitHub Issues threads I linked above and help the relevant teams diagnose/improve the problems.

I found it was really slow when assets were in debug. I generated a new rails app and copied the config from there and then merged in the settings from the old project I needed and it was super fast.

I used this command to get the debugger working on 1.9.3-p0.

    bash < <(curl -L https://raw.github.com/gist/1333785)

Not mentioned anywhere is that the PostgreSQL adapter now supports using the Schema Search Path without fistfuls of bugs and oversights.

I'm using schema separation for a multitenant application, and it looks like I'll be able to remove at least 4/6 hacks to make rails support that functionality.

Could you elaborate?

Sure. Basically, I have a schema called template_tenant, and a table public.tenants (ie. in the public schema). For each record with id n in the public.tenants table, I have another schema tenant_n cloned from and kept in sync with the template_tenant schema.

The result of this is that I have dozens of tables duplicated across n schemas. In Rails 3.1, the Postgres query cache, exec cache, and the identity map do not account for this, nor does table_exists?, and the connection pool would often check out a new connection without preserving the search path.

In Rails 3.2, most of these are fixed. The exec cache, query cache, and table_exists? now observe the search_path properly, and I believe the connection pool behaves better, though I haven't had time to confirm that. The identity map still doesn't understand, which I've filed an issue for: https://github.com/rails/rails/issues/4044

I'm really happy with this pattern for multi-tenant apps. It gives me a lot more confidence that no data leaks will happen than scoping queries, and the performance increase is significant with large datasets (unless of course, for some reason, you need to aggregate across tenants). It's nice to see rails supporting this better.

The sixth patch I had hacks support for iterating over multiple schemas to migrations. I add "schema_set :public or :tenant" to the top of my migration, and it runs either once on the public schema, or n+1 times on my template_tenant schema and my tenant_X schemas. It wouldn't make sense for this part to be in rails proper at the moment.

Could you? :)

Also, if it is not much to ask, I know this is not the topic, but could you describe how you make this kind of separation? I have the same requirements for isolation and your experience in the matter would be valuable. :)

Awhile back, I blogged how to go about this. Some of the hacks will not be needed for Rails 3.2, but the concepts are the same:

http://blog.jerodsanto.net/2011/07/building-multi-tenant-rai...

I remember reading your post some time ago, it really helped me creating a better view of this solution and is in fact our preferred method thanks to it. I still haven't implemented it for real and it would be great, if not much to ask you, if you could elaborate the impact of these changes in a post update? What do you think?

I haven't read sant0sk1's post -- it's probably the same idea -- but I gave an ultra-high-level overview in another comment: http://news.ycombinator.com/item?id=3374294

EDIT: This is my migration hack: https://gist.github.com/1f5e67b018d304dd1198 . Most of the SchemaUtils/Schema stuff is just wrappers around setting the schema_search_path.

Good to see that rails developers are progressively seeing the database as a powerful tool and not just as flat files with funny accents.

Uh oh, I haven't even updated my clients to Rails 3.1 yet. Definitely not good to fall behind more than a major point release or two with Rails. I have a very old client running Rails 2.3.x and it's always a pain when they need the occasional update and Rake doesn't even work anymore.

As I understand it, a lot of work has been put into Rails 3.x series so that it is easier to upgrade your apps.

It should be a lot easier to go from 3.0 to 3.1 or 3.2 and hopefully from 3.0 to 4.0.

I do hope they improve their policy on regression-tests and upgrade instructions. 3.0 -> 3.1 wasn't very smooth for me, the bugs were just more subtle.

My favorite bug during the upgrade: Someone decided (long ago) it'd be a good idea to store a Marshalled object in the session-cookie. One class serialized therein disappeared in Rails 3.1. Result? Any and all requests with a session-cookie bomb out with an unmarshall error.

tldr: Make sure to rename your session cookie before rolling out the upgrade. This logs out all users but is still more desirable than the alternative...

IMHO the better solution would not to store marshalled objects in the session cookie. Especially ones that you don't have direct control over. There can't be any reasonable regression-test or upgrade test that could test for this.

The Rails flash is actually stored as a marshaled object in the cookie, so this broke for everyone who happened to have something stored in the flash at the time of upgrade (which is a lot of people). It should have been caught before but it did get fixed in the latest release.

I feel your pain, I have a bunch of legacy 2.2 and 2.3 apps.

I moved all dependency management from config.gem over to Bundler, and haven't had a dependency issue since.

I wish I was still on 2.3. I made the jump and am now running 3.1, after putting off upgrading for so long. I bit the bullet because most plugins simply don't backport bug or security fixes. But hardly a day goes by that I don't wish I was still on 2.3.

Why ? I found 3.1 upgrade with all the asset compilation etc, a bit painful but 3.0 brought in a lot of nice changes. (Arel for one)

It's considerably slower than 2.3 and the API fell apart. It's this weird system that looks modular, but including any given module into your code will almost certainly fail due to weird dependency graphs or because some modules seem to only work when injected into certain classes. This makes it incredibly frustrating to debug.

Generally I like Arel, but the changed behavior around default_scope is maddening. And I've found several pretty big bugs that have been annoying to work around.

"Generally I like Arel, but the changed behavior around default_scope is maddening. And I've found several pretty big bugs that have been annoying to work around."

Can you expand on this?

Sure. I usually add a :default_scope with an order clause to my models, since it makes using :first and :last considerably nicer. If you want to do sort in the opposite direction, you can't just add .order to your calls (this used to override the default scope in Rails 2.3), since Arel will just translate that to the senseless "ORDER BY id ASC, id DESC".

The way around this is to use :reorder, but :reorder doesn't work the same way and can generate invalid SQL if you're prefetching included associations. So, you're left with :unscoped, but that's insanely insecure because it drops any foreign key scopes you had in place.

Likewise, it only normalizes identical order clauses if you use the exact same casing. This one is a bit more esoteric, but if your default_scope is to order by "id asc" and you reorder by "id ASC", Arel will produce "ORDER BY id asc, id ASC". If they match in case, however, it'll be normalized to a single clause.

I haven't found the time to attempt a proper fix to any of these problems yet.

@jonleighton committed a big refactor yesterday that might be a good sign here; looks like a lot of the old monolithic AR code is now new-style Concerns, more of a defined API, etc. If the root problem here is ARel itself, that won't help, but if it's really AR::Relation, maybe Rails is moving in the right direction.

We use user-defined functions in Postgres, and I have the same complaint - in theory, the move toward composable relations is great, but in practice, there are so many hitches and gotchas and prior assumptions and backwards-compatibility requirements that it's never clean.

It sounds like you're not using rvm and gemsets (or a similar tool) for your Rails 2.3.x sites. Rake should never "stop working".

http://beginrescueend.com/

Unless you're on a platform where you can't get gcc >4.0, so RVM is helpless, and you're stuck with just bundler. Then the rake/cron nightmares come..

In my project, "rake environment" took 11.5 seconds on rails 3.1.x, and 8.7 seconds on rails 3.2. (1.9.3.p0 for both).

I'm quite happy about this, as the initial development boot time is what's been bothering me most lately.

We see ~20% improvement with the same upgrade (on 1.9.3p0).

8.5s to 6.6s on my laptop (with SSD). Nice!

IMO, 6.6 seconds to boot the app is still too slow, but it's certainly progress. :)

Sure, but it's better than the 15 seconds on ruby 1.9.2 and rails 3. Things are getting better!

Seeing about 25% speedup here with just the Rails 3.2 RC1.

Combined with rspec-rails 2.8.0.rc2 (which includes some Rails 3.2-related changes), I saw my test suite go from ~2.5s (Rails 3.2.0.rc1 w/ rspec 2.7.x) to ~0.43s (Rails 3.2.0.rc1 w/ rspec 2.8.0.rc2).

Ruby 1.9.3p0 across the board.

Just upgrading to rspec 2.8 made my test suite run 10x faster. You might have David to thank for a lot of that speed up. :)

Certainly a LOT of it is rspec 2.8.

But there was a definite increase with Rails 2.3 as well. :)

I agree. 2.3 was much faster :-)

Nice tech talk about the thinking behind the new router from Aaron Patterson (@tenderlove) at MagRails.

http://www.youtube.com/watch?v=sEbobUPqhL4

https://github.com/rails/journey is the code for his router

Great talk, also.

Key/Value storage built in?

I love rails :)

Reply on news.ycombinator.com