Rails 3.2.13 - Performance regressions and major bugs
blog.bugsnag.com
blog.bugsnag.com
Like I the one about scopes that effected Github -- github quoted [this commit](https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5...) as introducing the bug, which is the commit meant to address CVE-2013-1854.
So, okay, bugs happen, even with security fixes, I can forgive bugs.
But others of those performance regressions... " handing that task to Sprockets instead of resolving internally, which seems to be the cause of the performance issues."
This doesn't sound like the unanticipated side effect of a security fix. This _sounds_ like an independent change they threw into the same patch release with four (four!) security patches.
Why is Rails putting security fixes and other non-security-related enhancements together in the same patch, so you can't get the security fixes without also getting the possibly buggy new features/refactors? Does Rails actually do this by policy? Was it a mistake?
Bugs happen, including bugs in security fixes, sure. But it's BECAUSE bugs will happen that you don't throw unrelated new features or refactors into the same patch release as a security patch, so the only way to get the security patch is to risk MORE problems from the other stuff too! Come on!
This isn't a security patch - a stable branch with deliberately cherrypicked fixes - it's just a plain old "point release." And as is to be expected with Rails at this point, point releases break apps.
As a primarily Django guy with an interest in Rails... is this hyperbole/overstatement or is this a common perception?
IANARailsDev, obviously.
This bug looks like it was caused by the interaction of Github's scoping with a small change to Rails to attempt to unify the query building a little and fix a potential security issue. Where queries are merged where they apply to the same column, but (probably unintentionally) they were not merged when one was a string and one a key, which happened in some circumstances with scopes. Would love to see a slightly clearer explanation of this bug - the article doesn't provide one.
They could (and usually do) isolate the security patches in a separate release with just minimal changes - at least that would make the process more manageable, but unfortunately in this case I don't think that would have helped, as the change was directly related to a change for security and just had unintended consequences.
In fact I think from their blogpost github merged the CVE patches separately after testing rather than using the update, so having a separate security release wouldn't have helped them either, it's just an unfortunate confluence of circumstances for them due to a gap in testing (perhaps on both the Rails side and the Github side). See
https://github.com/blog/1440-today-s-email-incident
The only lesson I draw from it is that perhaps Rails could slow down their release cycle a little, issue more RC builds, and encourage large users like github to try them out extensively in development environments (hard to persuade people to do this on small point releases though). Up to now they've done a pretty good job IMHO and I've not seen many issues crop up, particularly with security releases, which are usually straightforward.
That allows you to provide the security releases as a separate patch file so you can apply individually. You can skip maintenance releases that cause problems but still keep things secure.
https://github.com/discourse/discourse/blob/master/lib/freed...
The discourse folks who threw this together have an app (Rails-based forum software) that was hit very hard by the performance regressions.
https://groups.google.com/forum/#!msg/ruby-security-ann/o0Ds...
Unfortunately that wouldn't have avoided this issue, as the problem was actually with the small security patch.
Why?
1. Security fixes are released very quickly (good thing), but more often then not they break existing code (bad thing) - and while you _can_ wait for the next patch to fix those break points, you're left wide open since everybody can see what was broken and how to exploit it.
2. I'm not nearly smart enough with Ruby to apply 'band-aids' I read people write in Rails source code. I just don't have the time and energy to fix code in the actual framework when I need to be fixing _my_ code.
======
So there you have it; I'm bailing.
It was a fantastic ride, Ruby is a beautiful language, but Rails is just a clusterfuck (_for me_).
I need stability and predictable behavior. If that means releases every 6 month to 1 year, so be it.
So long and thanks for all the gems.
But thanks for the passive aggresive comment.
Sinatra, sequel and postgresql is a dream combo. I wouldn't use rails for a new project these days, I got bored of the un-navigable code a while ago. These patches and regressions are the icing on the cake.
Because your comment didn't exactly hint at it considering its completely binary assertion of "Rails or ASP"
> But thanks for the passive aggresive comment.
I'm very sorry you found it passive, that was not intended.
This will come out harsh, but I'm being honest: If you are not willing to fix/adapt/improve the framework you are using, you better not be using ASP.NET also (try a google search on asp.net issues), or any framework at all. In fact, developing larger more complex software might not even be possible without adapting frameworks for your specific problems.
That said, our run so far with Rails and Java (we are still using 2.3!!!):
- We have 38 patches applied to Spring
- 4 patches in apache santuario.
- One security patch on Grizzly (glassfish)
- Rails: One single pull request https://github.com/rails/arel/pull/174
Huh what? Stack Overflow with it's extreme load works quite well with ASP.NET.
>try a google search on asp.net issues
Okay, I did, what I am supposed to be looking at?
From what I've read on http://highscalability.com and the like, they love to tweak.
We've configured route registration a little bit specially* to speed up some common cases and we don't necessarily use every feature in the framework; but we're far from a proper fork.
ASP.NET MVC performs well enough that we get more bang from focusing on our own code basically.
*This amounts to registering our highest traffic routes _first_, and checking left-hand matches before Regex'ing (since we have a ton of "/foo/\d+/bar"-ish routes).
I generally prefer Ruby-based things myself - this post isn't trying to sway anybody to the MS side; just wanted to give a bit of credit where it's due and perhaps a bit of news to those who haven't looked over onto that side of the fence in a long time.
Trust me, I have been thinking about what you say every day for the past 5 days.
We haven't looked back since -- well, we have given an glance or two back since, and viewing the slow-motion car crash that is Rails today, we just feel sorry for those left behind that are discovering 'The Rails Way'.
Ironically, given DHH's comment on Rails about it being 'Omakase', I should point out that it translates as "I'll leave it to you" -- security, I'll leave it to you, software engineering, I'll leave it to you.
Food for thought indeed.
Remember -- Ruby to pose, Python for Pros.
This is one of the largest lies perpetuated on this site. If you know just enough of something to follow some tutorials and crank out cookie-cutter sites using Rails/Django/Asp/Grails/CodeIgniter/etc, but fail the moment you encounter basic ecosystem problems, you aren't a polygot. You're a liability to your team for not understanding anything well enough to work around common issues. You're probably curious and love to try new things, but the implied part of being able to proficiently wield whatever tool is correct, or at least on hand, is something escaping you.
If by polygot you mean you know some syntax, the basics of the ecosystem, and generic system design, algorithms, etc while expecting to dive as deep as necessary into any given environment, you're still failing the definition. This would be the point where you RTFS so you know where to override/monkeypatch/workaround.
I see this a lot on HN too: people who've experience some level of success (as pg would say the first thing you learn when you get rich is that there are many levels of rich : ) and who hence think they know it all about everything and can constantly try to diminish others.
There's a lot of negativity here but, thankfully, there are also others who are here to share, educate and learn.
If you want, you're welcome to use Rails 2.3 or older and you'll encounter many fewer changes. I imagine the 3.0 branch will have much the same stability once major development moves to 4.0.
It's quite possible that someone will pick up the baton, as there still are a number of production 2.3 apps out there, and porting to 3.0+ --- more a "port" than an "upgrade" --- is a real pain. But I'm not sure anyone has stepped forward yet, and until someone does, you're taking your chances.
Also you gain a _lot_ in security by obscurity and hipster points. I don't know about stability, haven't tried this myself.
Check out the mono project for details: http://www.mono-project.com/Mono_Project_Roadmap
3.2.11 and 3.2.12 were both security-only releases, and so fixes and patches that were not security-related have just been piling up for the longest time.
Seems like a "release often" mindset might be better and might ensure that things are actually being used. Sure, I know someone is going to blast me saying that bugs like this shouldn't make it into 3-2-stable at all, and since this is Hacker News so whoever says that knows they're ten times the developer that everyone on the Rails core team is. But the reality is that bugs make it into stable branches, and almost nobody is running against the stable branch (Rails core people are probably mostly running against the 4.0 branch, everyone else is running against the latest release).
To your more general point, while I don't disagree with your diagnosis of things piling up in stable, I'm also not a big fan of your proposed solution. It looks to me like 3-2-stable is getting filled with a lot of stuff that just should not there; increasing the release frequency is not going to solve this problem.
As for the rest - I think the issues in January has put Rails in the spotlight, and while scrutiny is certainly warranted, I think the extra pressure on the project over the past months to clear this stuff up has had its consequences.
We didn't, however, realize that "patch" releases also have release candidates. I'd recommend following the rails blog (http://weblog.rubyonrails.org/) which announces release candidate builds too.
It looks like this release was quite fat, for a patch release.
I added "check rails diff" to my deploy todo list.
EDIT: I just noticed that http://weblog.rubyonrails.org/ doesn't have a way to subscribe by email. I signed-up by email with this site instead http://blogtrottr.com, in case it helps other here.
But seriously, can we at least expect moderately stable releases? I do not think that's too much to ask, especially for a framework which has, in the past, achieved that.
Serious answer: a few months ago, some serious vulnerabilities were found in Rails. This apparently had a "blood in the water" effect on security people, who started dissecting the entire thing. Predictably, more vulnerabilities have emerged and rapidly been fixed.
All in all, it means that things are improving because it's getting a ton of attention.
One might argue that these performance regressions are unacceptable, and they are. I agree. Still, Rails is relatively young framework and these things happen. I expect any framework that is widely used will see some serious vulnerabilities in the next few months/years.
By the way, a good explanation of what these past Rails security issues mean is explained on patio11's blog (http://www.kalzumeus.com/2013/01/31/what-the-rails-security-...).
OT: I'm also writing a book on upgrading your Rails 3 app to Rails 4. With these rapid releases, a lot of things change and it's not always clear what has changed or what is new. Feel free to check it out at http://upgradetorails4.com/.
> […] handing that task to Sprockets instead of resolving internally […]
even remotely related to performance issues because of random routing? You're mixing two issues that have nothing in common.
2) The Rails release has performance regressions (i.e. in at least some scenarios is slower).
In situation (1) you don't want to add further slowdown so (2) is a bad thing.
Grandparent doesn't even blame his Heroku performance problems on the routing issue (although it is a plausible guess). Actually effect of the routing problem can be significantly worsened if even a subset of requests are slowed down.
Is that not the case?
Performance Regressions
There are performance regressions in 3.2.13 for both view loading and asset loading. Rails 3.2.13 changed the way assets paths are resolved, handing that task to Sprockets instead of resolving internally, which seems to be the cause of the performance issues.
Can you point me to the specific ones so I can look at them again?
I'll bump this up in my personal list; I should really learn AR better anyway.
I'm also sure that people are working hard to fix these recent security issues and perfs issues.
But as a non-Rails dev I can tell you one thing: with all the attention that Rails got lately I'm sure I'll never be learning Ruby / Rails.
I'm into Clojure right now. Next target is Go.
Sadly all these Rails exploits do have a negative effect for Rails: I'm not saying this to criticize Rails. I'm saying this because I'm honestly 100% sure I'll never ever be doing anything serious with Rails and I know I'm not the only one in this case.
So, Rails devs, fix this and fix this rather sooner than later because every exploit is missed opportunities.
That said, I don't really know if RoR has had more than normal or whether they're just more widely talked about. It does have a huge and vocal user-base.