Up to 12M websites may have been compromised via Drupal vulnerability
bbc.com
bbc.com
At times like this, I really wish there were more best practices to mitigate against these type of vulnerabilities.
One idea is using the remote-execution vulnerability itself to remotely patch the servers by passing the antidote as the payload. I am not sure if this would be legal.
The other is something what CloudFlare already does but doing it at the hosting-provider or ISP layer. It need not be as proactive as CloudFlare which can inspect HTTP requests for known exploits and blocks them. It could be something as simple as running scanners and routinely informing customers that their websites are vulnerable. Or better still, temporarily disabling the account or putting it into read-only mode, thereby forcing users to take action.
There are many best practices:
- Use version control.
- Have remote backups for the past day, week, month, etc.
- Apply security patches immediately (and have test infrastructure so you can make sure they don't break anything).
- Be able to rebuild a server from scratch w/ backups (configuration management really helps here).
- Subscribe to security mailing lists/RSS feeds/Twitter accounts.
If nothing else, use this situation to prepare for the worst on your own site; how would you respond if your site was hacked, your database in an unknown state, and your codebase potentially backdoored?
Once again, software is harder than it needs to be. Gary Bernhardt has been going into this in depth on Twitter lately, and I agree with him more and more as I pay more attention to the shitshow around us.
As someone who worked on Drupal for a couple of year in the days of 5/6 this was my biggest problem.
I don't think things have improved much.
Sadly, most Drupal developers and development shops either don't know about or don't care to take the time to build sites in this manner (instead of schlepping databases and file dumps all over the place, export everything to code)... but if you do, team-based/large project Drupal development becomes so much more sane.
The theme will contain php files but their number is relatively limited and is primarily templating related, it should be possible to find bad code here if you know what you're looking for, after all the theme doesn't handle authentication so can't exactly introduce subtle 1 character tweaks here to make the site vulnerable.
The database is more complicated unfortunately, besides simple privilege escalation and permission alteration which are relatively easy to find, drupal often allows php to be stored and executed from within the db. I can see if an attacker is particularly sophisticated a backdoor in there being harder to find, but I imagine there will be people working on tools to hunt that kind of thing in the imminent future.
I like the idea of deploying an antidote but that won't always work and I can see people not liking it. CloudFlare would be an example of defense by layers.
Another observation is about how the vulnerability is disclosed. I don't think this was handled properly here. Ideally you want to see a disclosure that allows users to take action without actually revealing the specific vulnerability. An extreme example would simply be to tell all Drupal users to take their sites down and give them enough time to do this before disclosing anything. Possibly a less broad action could be taken while still hiding the specific exploit. To release a patch revealing the exploit while many users are still open to an exploit is extremely problematic.
Scenario #1: Release patch with no advance notice. Takes time for people to take notice and apply. In that window lots (12 million?!) sites get hacked. That doesn't sound like a very happy outcome.
Scenario #2: Announce a patch will be released in advance. Get the word out. Give people time to hear it. You're not giving any specifics other than there is some vulnerability. That's doesn't help any hacker, every product has some vulnerability. There'll probably be a lot less than 12M sites hacked. Also consider closing the vulnerability in ways that don't directly touch the affected code to make it more difficult to reverse engineer.
Obviously if everyone knows of the exploit you don't have the time but often the way hackers hear about the exploit is through your announcement...
[EDIT: Thanks to aryx above I learnt there was a 5 day window in this specific case (An Oct 10th announcement for Oct 15th security update). I don't think that announcement made it clear enough what the consequences of not applying the patch immediately would be. I think this is something that needs more careful consideration in the future.]
At any rate, worse thing is simply announcing [EDIT: i.e. releasing] a patch that is not going to be applied immediately by a large percentage of your users and allows all attackers to attack those sites. I can't quite think of a worst approach; even never patching at all might be preferable to that.
They did. https://groups.drupal.org/node/445893
> You can tell everyone to take their site down between the release of the patch and the application.
Do you honestly believe businesses relying on Drupal for core parts of their business will do this? You are seriously naive.
> Depending on the vulnerability there may be a way of blocking it without specifically advertising the vulnerability (e.g. block it in unrelated code).
If there's a patch it's trivial to find out what it affects.
> I can't quite think of a worst approach; even never patching at all might be preferable to that.
No, that's just stupid talk because it assumes that no one else has known about the vulnerability before the announcement.
This quote is not about the people who knew about the vulnerability before the patch was released on Oct 15th, right? I'm sorry but I still think that if the outcome of the release of the patch is 12M sites hacked this isn't a good outcome. I wasn't aware of the 5 day window but maybe a bigger announcement should have been made. I never heard of the issue (and it wasn't on HN) prior to the release of the patch. The Oct 10th announcement wasn't proportional to the size of the issue.
It's trivial to find out what a patch affects but it's not necessarily easy to find out what the issue is unless the patch actually addresses the specific bug. I'm talking more generally than this specific exploit. Let's say there's an OpenSSL bug in AES/128 implementation. You could release a patch to the AES/128 code or you could release a patch disabling AES/128. The former is an open invitation to hackers. The latter still leaves a potentially large task of figuring out what exactly in the AES128 is the issue. I'm not saying this is always possible but I'm pointing it out as an option which should be explored (and I've never seen utilized). By all means just feel free to give the hackers the detailed attack vector as a lot of the latest releases/patches have done.
EDIT: So referencing this: https://www.drupal.org/PSA-2014-003
Oct 10th: Announcement of some security update without detailing the implications:
Oct 15th: Patch released
7 hours after patch release mass attacks begin.
Oct 29th: If you didn't patch within those 7 hours consider yourself hacked. Until this release this didn't really get many eyeballs.
Nov 1st: Story about scale of hacks hits mainstream media.
What? HN is suddenly the be-all and end-all of security announcements? I heard of the issue way before the release of the patch. Anyone that subscribes to announcements from Drupal has heard of the issue.
You say that 5 days isn't enough? What period of time is? A week? A month? A year? You can always find people who somehow miss the announcement.
> It's trivial to find out what a patch affects but it's not necessarily easy to find out what the issue is unless the patch actually addresses the specific bug.
It is trivial Take the recent POODLE attack for example. The rumors floating around points to it being an issue in SSL 3 and not TLS 1.0. That contained enough information for someone to preempt the actual announcement with the exact attack.
> Nov 1st: Story about scale of hacks hits mainstream media.
Who the hell cares about the mainstream media when monitoring issues related to software you administer? That's just negligence.
You're completely missing my points. HN is not the be-all and end-all. It's a proxy for the visibility some specific announcement gets. You must update or you will get hacked would have gotten noticed. A mild message about some upcoming unknown security patch, not so much. And yes, by drawing more attention you increase the risk of getting attention from attackers but in this case it doesn't seem like the right trade-off was made.
Given the specific scenario there are certain variables under your control. There's the timing and "volume" of the announcements. There's the timing and content of the patch. You are trying to set those variables to minimize the number of affected people. If you think this case (12 MILLION) was anywhere close to the minimum I think you're wrong. The period that is long enough is the one that minimizes the number of sites hacked, in this case 5 days from this non-announcement was obviously not enough. I don't use Drupal and I've no personal connection to this issue whatsoever I just judge it by the end result.
It's also absolutely clear there are degrees of disclosure for the specific vulnerability. Having a clear description of the vulnerability makes it easier for someone to take advantage of it. Your sample of one counter-example doesn't make any difference. I'm not saying you can always avoid someone taking advantage I'm just saying if there's a choice between making it easy and making it a little less easy you should chose the second. It's just like having a lock on your bicycle doesn't make it impossible to steal. It may cause the thief to move on to an easier target.
The unfortunate reality of the latter one is that it's going to break their site, probably, since we don't know what they run, and you can't just read only an entire account blindly without breaking something. We could do better there, but it will never be perfect in this environment.
I suggest that themes/plugins with 100% compatibility ratings should be auto patched too. Auto patching themes can be problematic because updates override changes you've made to the theme files. So my other suggestion would be to automatically create a child theme for every installed theme so that devs can easily update the parent theme and keep changes made to it.
[1] http://codex.wordpress.org/Configuring_Automatic_Background_...
And before someone makes a "joke" about PHP, they're using the PDO framework which supports bindParam(). Seems like woful incompetence in the Drupal codebase to me.
Exploit in detail: https://www.sektioneins.de/en/advisories/advisory-012014-dru...
The question is whether Drupal really needed to implement their own engine for this, though.
If your database supports it set PDO::ATTR_EMULATE_PREPARES to false. If not, I'd still take PDO's implementation over a million different implementations which other projects come up with such as this one.
Is 'elegance creep' a thing? I would be willing to bet money I don't have, that it's because someone thought it was more elegant and clever, and therefore just better.
I don't think bindParam will allow you to bind an array into a tuple , the only way to pass the tuple is by forming it as a string and adding that to the query string.
I'm using a Micro ORM (Insight Database, https://github.com/jonwagner/Insight.Database) that takes advantage of this. It maps a parameter of type IEnumerable<SomeType> automatically to a TVP. This ORM works brilliantly when you want to get everything out of SQL.
The problem here was a mistake someone made, not a fundamental support problem with the language.
This is precisely why people mock PHP developers. :/ So many don't even understand how the language f'n works.
It has nothing to do with passing security audits or withstanding attacks, there's not a security flauw in the way PHP handles this because PHP or specifically the PDO framework relies on the user to implement this themself. There obviously can't be a security flaw in something which does not exist.
A quick Google search suggests it is not at all obvious to many how to do a parametrised query with an IN-clause using PDO. The highest ranking answer is this SO post: http://stackoverflow.com/a/1586650
Having to iterate the array yourself adding the right amount of placeholders and binding individual values is secure but a lot of boilerplate. Escaping values in PHP and concatting them in the old fashioned way ought to be safe but everyone switched to parametrised queries for a reason: in practice it's often fucked up which leads to security vulnerabilities. The last one, the find_in_set trick, is a clever kludge but a kludge nonetheless.
People shouldn't need to roll their own way to do this because that's where unnecessary mistakes get made.
Alot of programmers can't do FizzBuzz either. If you can't figure out how to iterate and count an array on your own...
I'm sorry but I have 0 sympathy.
Plus, they might be able to figure out how to iterate and count an array but they might also figure out how to use implode instead which is less code and programmers tend to be lazy. And suddenly they've opened their app up to SQL injection because they forgot or are unaware they now need to do escaping despite using prepared statements.
And since their app might contain my data, I care about this and not just think "those idiots brought it upon themselves".
The only explanation for not doing that is ignorance and/or incompetence. Mistakes happen but to claim its a language problem is incorrect.
http://php.net/manual/en/pdostatement.bindparam.php
<?php /* Execute a prepared statement by binding PHP variables */ $calories = 150; $colour = 'red'; $sth = $dbh->prepare('SELECT name, colour, calories FROM fruit WHERE calories < ? AND colour = ?'); $sth->bindParam(1, $calories, PDO::PARAM_INT); $sth->bindParam(2, $colour, PDO::PARAM_STR, 12); $sth->execute(); ?>
So what you do is you write a function to convert the array to a series of ?'s for the IN clause/tuple and then iterate through the array to bind the parameters.
http://php.net/manual/en/pdo.constants.php
You can write "SELECT calories FROM fruit WHERE name IN (? , ?, ?)" and then bind the parameters as strings but this will only work in cases where the length of the tuple is known and fixed. If you need to allow for a variable length tuple then you will need to concatenate the query string yourself.
Dynamically rewriting the SQL isn't going to be tractable in all cases - doing `IN (NULL)` might be a valid value - and throwing an exception is poor form for an actually valid case.
What part of you needed to write a function was unclear? o.O
Counting an array is a safe operation.
No user input involved.
Of course you can. Everyone here is saying you shouldn't have to. This is the perfect type of thing to move into the db library. That makes it much safer because you don't have to hand-roll the same stupid code in 50,000 places (and possibly fat-finger it once).
Somehow this vulnerability went over the radar.
What is interesting is that based on our own data, we started noticing attacks around 8 hours after it was disclosed and we shared some of the payloads being used here:
http://blog.sucuri.net/2014/10/drupal-sql-injection-attempts...
The worrying thing as always is that few people upgrade these instances once they are launched and operational.
In the Drupal world, there are some similar managed Drupal hosting providers (e.g. Drupal Gardens), but they're much less common. I wonder why.
I have to link back to my earlier comment about the key takeaway here[4]—not just for Drupal sites, but for anyone who operates any site on any server. You can't afford to let your site sit unmaintained if you value the information within; and if you build sites for other people, you have to convey the importance of that to your customers... 'With great power comes great responsibility' and all that jazz.
Your site is either currently broken, or will be someday; it's not about making 100% secure code and servers (you strive for that, of course); it's about your response once something happens (e.g. a security patch is released).
[1] https://www.getpantheon.com/
* Look through the menu_router table for suspicious looking entries. * Look at users & user roles, anything new? * Look for scripts and executibles in your public and private upload files directory.
If you really don't know drupal and it's a new project, start from scratch (New Database but you're code should be safe assuming your repo isn't stored on the same server and there's no write to repo access on server)
$ find /var/www/blah.com/htdocs/ -iname "*.php" -mtime -1 -printhttp://blog.sucuri.net/2014/10/drupal-sql-injection-attempts...
In there you can see the type of backdoors being added (generally fake users with admin-level privileges).
Were there compromised sites? Sure, but I would be surprised if it were more than an order of magnitude less than what the bbc reported. That is still a lot of sites, but not a monumental cluster-fsck that the aftermath is being made out to be.
In fact I've seen 0 evidence of even one site being compromised.
Any links with substance?
It is a bit of a culture shock if you're using to spewing your PHP all over everything ala other lesser blogging tools, but once you've got your head wrapped around how it works internally, you can be very productive and do things that would be difficult/impossible in other systems with almost no code.
Hopefully the code quality coming from an average Drupal shop will increase.
Now, can we fix the culture behind those other lesser blogging tools too, please?
I haven't used Drupal because the job postings usually advertise super low wages and I think I saw the code base and was totally freaked by how messy the modules and things were.
I won't even consider building something with Drupal unless it was secure but I'm sure this makes sense only in hindsight.
Right now WordPress is way more popular though.
Also, Drupal is often coined a CMS. If you google "drupal is not a cms" you'll find a number of people referring to it as CMF - content management framework. It really allows you to build a custom CMS for a specific purpose rather than customizing a CMS for a specific purpose. It may sound like semantics, but if/when you have a few Drupal sites under your belt you really get what they mean.
[edited for clarity]