I can't get to the article at the moment, but I'd love to know why they failed to update their app especially since it handles financial transactions. I had several apps to update and the process took very little time and effort.
It was only a day or so from when a patch was available to when a public exploit was everywhere. Maybe they were on vacation, maybe the site was built by contractors who have since moved on, maybe they don't read hackernews or other programming news sources. I expect to see similar stories in the coming weeks.
Having said all that, if our upgrade holdup had taken any longer than it did, we would have implemented one of the mitigation strategies.
I don't know that that's what you're saying you did but we need to be glacier-blue-ice-clear about this. Nobody gets to wait on bugs like this. You patch or workaround immediately or, most probably, you shut your app down.
EDIT: I guess that qualifies as a mitigation strategy, but when I said that, I was talking more along the lines of the patches, or like another person I know, even more dramatic steps like forking Rails. There are regressions in the 3.2.x updates since 3.2.9 that affect some sites.
Bottom line is that there was a lot of bad timing here that sucked up a lot of time in securing a Rails site.
I wanted to be careful not to point a finger at you; it's just that this is exactly the kind of crazy mistake I can see a web startup making.
Obviously, the timing sucked, but nobody had any control over that.
https://gist.github.com/4512579
http://dev.metasploit.com/redmine/projects/framework/reposit...
You also cannot leave 6 years of vulnerable Rails-versions up on rubygems.org without even backporting your patch (yes, they're still up there now).
It makes no sense to blame the users of a web-framework, most of which are not security experts, for not responding instantly with the perfectly correct procedure to an event of this magnitude.
Put the blame where it belongs, on the idiots who "handled" the issue by releasing a "recipe to exploit"-Advisory. The same idiots who ramble about "not breaking old apps" when asked why the vulnerable gems are still up on rubygems.org and continue to be quietly installed by any Gemfile referencing them...
A bit more creativity should be allowed when handling a vulnerability of this magnitude.
For example, publish a patch that escapes all input in curious ways, presumably to prevent SQL injection. Pad it, obfuscate the actual fix with code-noise, make it annoying to read. Then release it as some handwavy, semi-plausible "follow-up" to the previous SQL-injection, urging everyone to upgrade in small caps.
Yes, a handful of blackhats will see through the bluff. However, it will likely be the exact same blackhats that already had discovered the issue after the initial SQL injection advisory.
This would have given the Rails-community a head-start, rather than unleashing every script-kiddy on the planet at once.
And then downplaying the severity by not encouraging developers to take the follow-up patch seriously?
That's a flat out terrible idea.
That seems like something that would ruin any future credibility of the project. No further security vulnerabilities could be trusted as being accurate (and any further patches would just be begging for additional scrutiny).
Also there is no "downplaying" in declaring any kind of problem as a remote SQL injection vulnerability. That is still obviously urgent enough for everyone to patch immediately. Yet it doesn't attract blackhats in the same way as blarting "remote code injection".
There were people on this very site who were commenting about whether they should concern themselves with this patch (initially because people erroneously attributed the vuln to SQL-I, and then later because they "weren't using XML anyway")
Those kinds of things happen when you don't clearly describe what a vulnerability is, and when you try to mask how big of a deal something is.
This was a huge vulnerability. It was critically important that everyone running a Rails app fix it immediately. Shouting that from the rooftops was absolutely the right approach. Cloak and Dagger bullshit to try and hide that is unequivocally a bad idea.
I disagree with Moe that labeling the latest bug as an SQL-I instead of RCE is a good strategy to ward off blackhats.
For me this very article (bitcoin exchange being hacked) demonstrates that despite the harsh wording, the advisory didn't reach everyone in time. Not even those who should really care (like bitcoin exchanges).
I still think the explicit disclosure did more harm than good, by drawing maximum attention from the blackhat-community without really improving the reach amongst the oblivious. I still think a "staged" disclosure might have worked better, even at the risk of the timeline being short-circuited by a malicious party spilling the beans early.
However, there's little point bemoaning this particular baby or bathwater much more now.
I'm more interested in the steps that the Rails-team will be taking to lessen the blow by future security incidents. As I've said in another thread I'd be in favor of an optional (opt-in) kill-switch, to be triggered only in drastic cases like this one. Perhaps that is a point where we can agree again - otherwise we'll have to part in disagreement. :)
I'm all in favor of giving people more time to upgrade, but this issue was very unusual. Multiple groups were all finding it at the same time and who knows when the first "full disclosure is always the right answer" group would have started singing? Like it or not, full-disclosure people exist in the world, and responsible-disclosure people need to be aware of their existence.
If just one person/group had found this, RoR could have done a better "get ready to upgrade next Tuesday at 6am" followed by a "here is the patch, details to follow tomorrow" on Tuesday at 6am.
What needed to happen is what happened: a patch and a series of workarounds were produced as soon as possible.
Nobody got to choreograph this one. The control you think the Rails team had over this, they did not have.
Nothing about this situation was good, but hiding it would have only made it worse. When you're serving shit sandwiches for lunch, it's best to let everyone know that's what's on the menu. End of story.
I actually remember this incident (albeit blurredly, hell, was that really 10 years ago...).
I dug up a few snippets[1] and I think the main sources of the animosities back then were certain regressions caused by privsep (not a hassle-free change) and a bit of ego-clashing (Alan Cox, Theo).
Your summary of the events may be about right, but is "not wanting to dance to someone else's tune" really a good argument against an attempt at responsible disclosure?
[1] http://www.baylisa.org/pipermail/baylisa/2002-June.txt (at the bottom)
I don't know why people weren't taking a hard look at the params processing code path in Rails for 6 years, but they clearly weren't. The SQLI bug from last week pointed people at that code, it got its first real shake, and that's all there is to say about it.
You pull off the plug ASAP.
Better have customers unable to use the service for one day than have the customers lose their money while you stumble trying to figure out how to patch it. This is elementary for any mission critical system. Just imagine your neighborhood nuclear plant delaying the insertion of the control rods during a meltdown because it has to wait for some shinier parts to arrive.
The fact remains though, that this exploit was _so_ severe, that, depending on how attractive a target you were and how disastrous it would be for you to be compromised (financial services? High on both scales) -- it would have been a better choice to _take your app down_ until you can fix it, rather than leave it up with the vulnerability. The vulnerability was "an attacker can run whatever code they want on your server, and they can discover that you are vulnerable by cheap automated port scan." It's as bad as it gets.
But if it was somewhat important, kill the application servers until you can be sure to have upgraded properly.
When systems are designed with security in mind, rather than simply throwing together a web/sql application, people put a lot of time and effort into constructing barriers to protect the integrity of their data.
Take for instance the CACert certificate authority. They designed their system in a way where the master key that signs certificates is stored in a computer that isn't connected to any network. The actual servers then talk to this computer over the serial port using a carefully crafted API when they actually want a certificate signed. This means signing certificates is slow and the key is inaccessible. So if all their servers got remotely compromised, a hacker would never be able to get the key and at best would probably be able to sign a short list of certificates before being detected.
Other people have suggested that this black box should store public/private key pairs generated from the user's password for each user on the exchange. So when a user signs up for an account on the exchange, Javascript code generates a private key from the user's password, client side. The corresponding public key is sent and stored in the offline transaction signing box. Whenever the user wants to initiate a withdrawal, the transaction signing box creates a random number that needs to the signed with the private key that corresponds to the public key it has on store. This way an attacker need to compromise a server and install an eaves dropping application that replaces new users' (or existing users changing their password) real public keys with its own. Just breaking into the server wouldn't do the attacker any good at first.
There is really no excuse for exchanges not using cold storage.
It's like a CA that creates the certificates in PHP right there.
You need the seperation and you need to closely monitor and control the transactions requested from the web interface to detect any fraud or misuse.
I can dream up architectures which limit manipulations, require the user to constantly type in passwords, etc. A company not security conscious enough to update is unlikely to have done that. But suppose they did, what happens? EVEN THEN you can turn the website into the digital equivalent of an ATM skimmer, and steal money. That is, of course, assuming that I have not managed to turn shell access into some more direct compromise of your whole network.
Here is the moral. If you're directly handling money or a money equivalent on your website, and someone has shell access, they will be able to steal from you.
That said, if the backend is also vulnerable to the same or another exploit, that's not going to buy you much.
It's a forum post without any details - your question is raised, but has (so far) not been answered.
Because these exchanges focus more on promoting the "world changing" ideology than they do on taking their users' money seriously.