Drupal.org compromised | Hacker News Reader

Drupal.org compromised

drupal.org

140 pointschaosmachine81 comments

A copy of the email (slightly different from the notice on the site):

Easier to read: https://gist.github.com/sergiotapia/d241db48ec4f31dfd0fe

Dear community member,

We respect the privacy of your information, which is why, as a precautionary measure, we are writing to let you know about an incident that involves your personal information. The Drupal.org Security and Infrastructure Teams have discovered unauthorized access to account information on Drupal.org and groups.drupal.org. Information exposed includes usernames, email addresses, and country information, as well as hashed passwords. However, we are still investigating the incident and may learn about other types of information compromised, in which case we will notify you accordingly.

This unauthorized access was made via third-party software installed on the Drupal.org server infrastructure, and was not the result of a vulnerability within the Drupal software itself. This notice applies specifically to user account data stored on Drupal.org and groups.drupal.org, and not to sites running Drupal generally.

We have implemented additional security measures designed to prevent the recurrence of such an attack, and to protect the privacy of our community members.

The next time you attempt to log into your account, you will be required to create a new password.

Below are steps you can take to further protect your personal information online. We encourage you to take preventative measures now to help prevent and detect the misuse of your information.

First, we recommend as a precaution that you change or reset passwords on other sites where you may use similar passwords, even though all passwords on Drupal.org are stored salted and hashed. All Drupal.org passwords are both hashed and salted, although some older passwords on groups.drupal.org were not salted. To make your password stronger:

* Do not use passwords that are simple words or phrases * Never use the same password on multiple sites or services * Use different types of characters in your password (uppercase letters, lowercase letters, numbers, and symbols).

Second, be cautious if you receive emails asking for your personal information and be on the lookout for unwanted spam. It is not our practice to request personal information by email. Also, beware of emails that threaten to close your account if you do not take the "immediate action" of providing personal information.

For more information, please review the security announcement and FAQ at https://drupal.org/news/130529SecurityUpdate. If you find any reason to believe that your information has been accessed by someone other than yourself, please contact the Drupal Association immediately, by sending an email to password@association.drupal.org.

We regret that this incident has occurred and want to assure you we are working hard to improve security.

Thank you, Holly Ross Drupal Association Executive Director

Easier to read: https://gist.github.com/sergiotapia/d241db48ec4f31dfd0fe

Thanks. I'll put that link in my comment.

I don't know if they've invalidated my password, or if the attacker has already accessed my account, but I can't login with my details.

I can't help but think that they would be better off just doing an UPDATE users SET password='';, and relying on the forgot-password functionality to let users get access again

edit I received my forgot-password email after 4-5 minutes waiting (their servers are under quite a bit of stress right now .. understandably) Once I was logged in via the forgot-password link, everything was quite snappy. Just give it a few minutes and it'll reach you too, then you can reset your password to something random. (I can't recomment 1Password[1] enough)

[1] https://agilebits.com/onepassword

Also not able to login. I ran the forgot password functionality but their FAQ says it could take up to 15 min. to receive the email.

It took me about 4 minutes to receive the email - once I got it, the servers were quite responsive. Just give it a few minutes ... it'll get to you eventually

The servers are responsive because all the rest of us are locked out, still waiting on our reset email :-/

It's been 20 minutes since I submitted the reset, and I still don't have the email.

Edit: got email after 40 min.

Our email systems are somewhat loaded from the email announcement so it might be a little slow. Things are slowly improving in speed however.

I think they already reset all passwords, because I'm in the same boat. I put in a reset request, which was accepted, but didn't yet get the email with the reset link.

I'm guessing whatever mailing queue they have set up is completely back logged right now until it finishes sending out the general account compromise email (displayed elsewhere in this thread). At least, that's my best guess. I haven't gotten that email yet.

Same here, I know I used one of three passwords, and I'd like to know which one I stupidly used.

I linked to this in my edit, but it's worth re-linking here: https://agilebits.com/onepassword

Use it. And never look back :)

I much prefer lastpass, and keepass for everything else.

Ditto. I'm unable to login with my credentials, and have yet to receive the password reset I requested (although I assume they are getting a lot of those right now, so it may be queued). All these compromises sure are getting tiring. I've already been hit with Evernote, LivingSocial and Linode so far this year (that I'm aware of).

Your password was scrambled, which is why you cannot login.

> UPDATE users SET password='';

That would be an extremely bad idea. Never set pass to empty on Drupal 6 sites.

  - Do not use passwords that are simple words or phrases
  - Use different types of characters in your password (uppercase
    letters, lowercase letters, numbers, and symbols).

http://xkcd.com/936/

http://arstechnica.com/security/2013/05/how-crackers-make-mi...

I have a genuine not rhetorical question.

Use brcypt: http://codahale.com/how-to-safely-store-a-password/

Am I correct that this advice wasn't obvious in the early years of Drupal.org?

If so, what ought a site like Drupal.org do, today? Should they update the password storage to use bcrypt, and require all users to update their passwords? i.e. Proactively do in a constructive way, what they'd otherwise need to do in a crisis mode?

If so, why don't more sites do this? Is it ignorance? Inertia? Or concern that they'll lose users who can't be bothered?

When moving from a weak password hashing scheme to a stronger one you can keep old passwords around and just hash them again with the new system (that is, run your existing hashes through your new hashing function), and mark them as using the twice-hashed scheme. Your password comparison function can have a special case to use the twice-hashed scheme when comparing passwords on login. Users are migrated to the new scheme upon updating their password.

This is the approach which Drupal supports for migrating from Drupal 6 (which uses unsalted MD5) to Drupal 7 (salted SHA-512).

So if I understand correctly, they could use the same approach when updating to do what they really ought to -- use bcrpyt?

As a result the very oldest pwds would be thrice encoded -- from unsalted MD5 to salted SHA-512, and run that through bcrpyt?

I did some very light digging, and I found out that Drupal uses a stripped down version of PHPass.

"A cut-down and reworked version of phpass (supporting the portable hashes only and requiring PHP 5+) has been integrated into development versions of Drupal leading to the Drupal 7 release, after a lengthy discussion and many proposed patches against various development versions of Drupal. There's a notion of upgraded hashes - these are phpass portable hashes of md5() hashes (which were used by older versions of Drupal), with the final hash encodings prefixed with a "U" (for "upgraded"). A more recent lengthy discussion has resulted in Drupal 7 switching from MD5 to SHA-512 for the underlying cryptographic primitive in phpass' "portable" hashes (making them less portable) while preserving "read-only" support for the MD5-based portable hashes. There was no valid technical nor security reason for this change, so it was made purely for "political" reasons. Drupal 7's SHA-512 based phpass-like hash encoding strings use "$S$" as the hash type identifier.

There's also a module for Drupal 5 & 6 that makes the original phpass available with those versions of Drupal, including support for the more secure but not nearly as portable CRYPT_BLOWFISH and CRYPT_EXT_DES hashes."

SOURCE: http://www.openwall.com/phpass/

While MD5 is insufficient, SHA-512 (default D7) and CRYPT_BLOWFISH offer very good alternatives. Overall, from my understanding, the biggest issue with Drupal is websites that are either running an old version, or used to run an old version but never forced users to reset their password.

I hate when a site doesn't let me delete my account. Apparently, drupal.org hasn't been able to implement this feature in more than 11 years already:

https://drupal.org/node/8

> https://drupal.org/node/8

While it was a long running thing that the Drupal product couldn't let users delete or disable their own accounts, the issue you linked was resolved[1] in 2009 (more than 4 years ago) and made it into Drupal 7.

Unfortunately, Drupal.org is still running on Drupal 6 and its upgrade to Drupal 7 has been met with repeated setbacks and delays[2] with no ETA[3]. Definitely unfortunate.

Edit: looks like they're accepting requests for account deletion via password@association.drupal.org for the time being.[4]

[1]: https://drupal.org/node/8#comment-1188824

[2]: https://association.drupal.org/node/17738

[3]: https://association.drupal.org/comment/4263#comment-4263

[4]: https://drupal.org/news/130529SecurityUpdate#comment-7475090

> Unfortunately, Drupal.org is still running on Drupal 6 and its upgrade to Drupal 7 has been met with repeated setbacks and delays[2] with no ETA[3].

In a nutshell, that line sums up everything that's wrong with drupal.

Respectfully, there are vastly better examples of "everything that's wrong with Drupal".

An unwillingness to push through an upgrade to a site containing millions of user accounts and tens of millions of pages of content before it's ready is, to me, an example of what's right with Drupal.

One can't delete their account on Hacker News either. Just sayin'. :p

Interesting, I didn't know. Is there some good reason for it?

From a programming standpoint it adds a lot of complication (e.g. all of a sudden you'd have posts owned by NULL, and even if you deleted the posts, what about replies to those posts, etc).

Well, yes, but that's really just excuses. Forums often remove the account and profile information and leave the content created by the user intact. That's a good start in my opinion.

Still extra work. My time is not infinite, and I would rather spend it on things for people who actually use my site.

To help you decide your priorities some countries now mandate this by law. Eg. all of the EU.

Given the amount of effort startups put into getting accounts, correlating their social media accounts and harvesting user data, and run aggressive analytics, I find it a bit disheartening that the comparatively minimal effort necessary to let them leave is ever considered a waste of time.

I shouldn't be considered a fly trapped in amber just because I happened to want to try a site out, particularly when so many require an email address or facebook account to sign up or sign in (and in some cases, to find out what it is the service does.) Hashing passwords is extra work too. Common courtesy towards users should count towards the minimal amount of work necessary for a site. It's not as if the users who want to leave, now want to stay because you want to hold on to their data.

Drupal 7 lets the admin choose any option for that, remove all content, leave it as anonymous etc.

Yeah, but the main Drupal website doesn't run on 7.

If only it was just Drupal. I went through the ~80 or so accounts I have logged in LastPass, including many I haven't used more than once, and the sites that allow deleting your account are a small minority. It is annoying to say the least and hold me back from opening any new accounts unless I'm pretty sure I'll use them.

> Malicious files were placed on association.drupal.org servers via a third-party application used by that site.

I wonder what 3rd party app this was...

Me too..

This is one of those cases where I completely forgot that I had an account, and now it has been compromised. I'm scratching my head wondering what information/password I had associated with it.

Makes a case for actively destroying accounts on services that you're no longer planning on using.

Or not sharing passwords between services.

You're right, though I don't share passwords between services, I'm curious just the same.

Preferably both. Not sharing passwords is the obvious one, but if I stop using something I'd like to be able to delete everything related to me (except perhaps logfiles or whatever else needs to be retained for operational reasons up to a few weeks). And without having to call someone on the phone please or email support@ please.

Or getting rid of passwords as a mechanism entirely.

Not that I have a solution, just whining :)

Mozilla Persona, I will suggest.

Moderately miffed that the email they sent out notifying how THEY allowed my password to be compromised included a lecture telling ME how to construct a strong password. Not the time, or place.

Besides, they claim it was salted, so it shouldn't really matter at this point whether my password was "Password123" or "@DJDF*$@!(DGEWGIRGHdfhEWROighMMMM...PIZZA".

> Besides, they claim it was salted, so it shouldn't really matter at this point whether my password was "Password123" or "@DJDF*$@!(DGEWGIRGHdfhEWROighMMMM...PIZZA".

Salting a hash isn't a license to use incredibly weak dictionary passwords like "Password123". All it does is prevent against rainbow table attacks, where an attacker uses a pre-computed list of password-to-hash combinations to determine if you used one of them, and slightly slowing down a brute force attack. Due to the speed of brute force attacks these days, rainbow tables aren't really in vogue anymore anyway, so it winds up only buying a bit of time.

Ars Technica had a recent story about this: http://arstechnica.com/security/2013/05/how-crackers-make-mi...

> Salting a hash isn't a license to use incredibly weak dictionary passwords like "Password123"

Agreed.

> All it does it prevent against rainbow table attacks

Salts do prevent rainbow attacks, but they also assist in brute force attacks. The article you reference was over passwords that had simply been MD5 hashed, which is about as computationally significant as doing nothing at all these days . Even in the ridiculous case of using MD5 for hashing, though, a 16 byte salt raises the cost of brute forcing a password substantially.

My point isn't that strong passwords aren't important, it's that if the salt + hash procedure that was used is adequate, then the strength of any individual password becomes insignificant in relation to the strength of the password + salt. (Well, my real point was just to vent, but anyway...)

I agree that it does slow down brute force attacks (I did say as much in my comment: looks like I was finishing up the edits to my comment while you were replying), but not substantially, particularly when the password is already weak. The salts themselves are stored in plaintext, so while it prevents the ability to use prior work on future passwords, weak passwords will be compromised just as fast as if there were no salt at all.

"My point isn't that strong passwords aren't important, it's that if the salt + hash procedure that was used is adequate, then the strength of any individual password becomes insignificant in relation to the strength of the password + salt."

I'm pretty sure that's not true any more. Modern gpu based password crackers like Hashcat will rip through their lists and algorithmic modification of those lists just as quickly with no salt, 2 byte salts, or 16 byte salts. The salt adds no protection at all against a brute force attack - especially wordlist-based brute-forcing (at least not against an individual password - it _does_ mean that the first time you find one account using "password123" you don't automatically have all of them).

This article: http://arstechnica.com/security/2013/05/how-crackers-make-mi... explains it nicely - it kind of muddies the water by concentrating on a list of plain MD5 password hashes, but Hashcat is just as happy to rip through any of over 30 different password algorithms including just about any variant of MD5+salt and SHA256+salt (http://hashcat.net/oclhashcat-plus/#features-algos). (Astute observers will note that list _doesn't_ contain "slow hashes" like bcrypt/scrypt/PBKDF2 - that's why Coda's "just use bcrypt" is still good advice for developers storing passwords.)

The only sensible password advice these days is:

1) Where possible, use two factor auth - make sure the email account that receives password resets is tfa secured, or all your passwords can fall to an attack on that email account.

2) Always use password storage/generation software (1Password/KeyPass/LastPass/whatever), and use a long but memorable passphrase for it.

Once you've chosen to do that, it them becomes easy to follow the next rules:

3) Never reuse passwords - generate a new random unique password for every login.

4) Use long complex passwords - at least 15 characters including upper/lower/digits/symbols. (I use 25chars except to the few passwords I need to type manually regularly - 1Password can't autofill my AppleID password on my phone or iPad - so I've got a 15char password in my head for that).

5) Recognise that some passwords are more important than others - your online banking, PayPal/eBay, Amazon, AppleID and/or Google Account are the obvious ones, but also take extra care with domain registrar accounts - if I can change your DNS, I can update your MX record, and bam! I've got your password reset emails. For the same reason your dns hosting is a critical account, if I can update the MX record on your webhosting cpanel account or your linode/rackspace/whoever server - I can get all your password reset emails. Treat all these passwords with extra caution - don't type them into "untrusted" computers.

So ya, I pretty much have to agree with everything you said. Salts have negligible impact on brute-forcing an individual password, but at least slow down the cracking of an entire database of passwords (as in the arstechnica article). I was definitely too general in my mid-vent statements there...

Regarding bcrypt/scrypt/pbkdf2... amen. Currently every system I maintain uses bcrypt. Prior to the *crypt slow hashes, I used a salt that was a combination of something stored in the db and some in-code transformations, and then pumped through a ludicrous number of sha-512 rounds. The salt generation technique I used was probably cryptographically naive, but the intent was to require both the data AND the code to be compromised for the hacker to have a fighting chance. Anyway, I still have no idea what I'm doing, which is why I [use bcrypt, use bcrypt, use bcrypt](http://codahale.com/how-to-safely-store-a-password/). :)

I'm not convinced by their salted claim, I thought Drupal 6 (what drupal.org is on) only did MD5() for passwords? Drupal 7 has the more securely hashed passwords.

Drupal.org runs D6 with the phpass module, basically the stuff that went into Drupal 7.

FWIW, a member of the Drupal community has made a pretty sweet Drupal / Mozilla Persona module (http://see.drupalpersona.me/), if you run a Drupal site and don't want to store password-derived data in your database at all.

Not saying that 2factor / persona auth modules aren't the eventual way forward, but the worst thing to do is to run out and do something like adding modules to your site. Installing modules that have any relation to authentication or security should be postponed until a careful review of the code and its consequences can be completed.

I am surprised noone mentioned Supergenpass. It was a bookmarklet originally but now there are browser extensions, mobile apps, a HTML+JS page you can store locally. It takes a master password which never leaves your machine and generates a hash based on the master password and the domain. Makes it superb easy to not share passwords and yet be able to log in on any computer.

Drupal.org compromised. If I visit drupal.org, will I experience the compromise firsthand via some zero-day exploit? A situation like this really calls for an independent site to make security-related announcements from where you can reasonably trust the independent site to not have been affected.

The site isn't being used to distribute an exploit. There were problems in the past and the page linked in this story describes what happened and what members of drupal.org should do to protect themselves.

I didn't make my point very clearly, and I think it can be summed up with an offline analogy: "Toxic nerve gas spill in Disneyland. To learn more, visit the Information kiosk inside Disneyland."

You made your point, but I blame the ambiguity on an overly brief title and no context in the original HN post. If the title said "User information on Drupal.org compromised" that would be more accurate and let you know that you probably don't have to worry about a browser-attack-zero-day. And, of course, if you're the kind of person who worries about a browser-attack-zero-day you're probably also the kind of person who has a Virtual Machine running a guest Tails linux with no plugins and javascript disabled so that you can visit sites like this without worrying.

Going back to your Disneyland scenario, a closer analogy might be "An attacker stole souvenir photos of visitors to Disneyland. To learn more, visit the information kiosk at the front entrance of Disneyland." I say photo because it's something somewhat private (like a hashed password) and I moved the kiosk to the front door of disneyland (i.e. outside of where new photos are taken) because people who want to learn more don't have to set a new password on Drupal.org.

They don't seem to mention this, but I'd say everyone that uses d.o git repos should definitely verify their set of authorized keys.

Definitely, as well as check your repos for malicious code. Unlike GitHub, Drupal doesn't send out a notification when a key is added, edited, or removed.

According to the OP, the compromise happened because of malicious software being placed onto association.drupal.org via a third party app used by that subdomain, and not because of a vulnerability in Drupal itself.

So...it'd be nice to know the details of what this third party app was and also, some basic details of the configuration of association.drupal.org. Not anything specific, but rather, how is the subdomain stack different than the one used on drupal.org?

Definitely. We've been doing a ton of investigation into the issue and are trying to get all the details right before we make a detailed technical post. There are already some issues in the queues of some modules and patches committed to others that would help prevent this kind of problem in the future on drupal.org and other sites. We basically got to a point where we wanted to let users know their information had been compromised before we got to the point where we're satisfied with the details of the attack to be able to fully share what happened in the attack. There will be at least one more technical followup in the future and possibly more. Those followups will get into more details of the class of problem we faced and what we think can be learned from the event.

I don't get why passwords are still not salted 100% of the time at these higher profile OS vendors. I was working on a clients Wordpress. And i never really touch it. To my amazement i found out I could simply md5 (new password) inside the users table to change my forgotten user pass. Wtf?

This is for backwards compatibility. If you checked it again after you login, you'd find that it is not an MD5 password anymore.

WordPress uses salted and hashed passwords with the phpass library, and has since version 2.5, released in 2008. Prior to that, it used MD5 passwords.

As part of the conversion process, it detects an MD5 password in the database on login, and then hashes the password to the newer salted mechanism, overwriting the MD5 version in the database.

While changing my password:

Drupal.org Site off-line The site is currently not available due to technical problems. Please try again later. Thank you for your understanding.

It looks like they are making some update...

for a minute i thought i will have to go check all modules downloaded or something like what happened to the rails gem repository. At this point they don't know yet, but hopefully it does not boil down to that.

Yup, I just got an email.

And that's why you go with Wordpress...

Why?

Because Wordpress has the most secure codez.

Really? Wordpress uses md5 by default, which though it's salted and hashed several times it's still far less effective than the phpass in use on drupal.org and the derivative in use on D7. http://codex.wordpress.org/Resetting_Your_Password scared me when it suggested that you reset your admin password by just using an md5 generator and doesn't tell you to change your password AGAIN once you've reset it with a single md5. This scares me too: http://codex.wordpress.org/Updating_WordPress#Automatic_Upda... auto update requires web-write to your whole site. No wonder it's such an attractive target to build a super-botnet: http://technorati.com/technology/it/article/wordpress-under-...

WordPress uses phpass since 2.5.[1]. You can reset you password using an md5 generator, but that is due to the backwards compatibility in WordPress. In order to provide seamless upgrades to the new hashing mechanism, they hash is not updated until the user authenticates again. The password is confirmed against the old hash, and if it matches, the password is re-hashed using the new hash and saved over the old hash. So if you have a password stored in the md5 hash and then authenticate once, you are updated to the new salted hash.

[1] http://www.openwall.com/phpass/

So http://codex.wordpress.org/Function_Reference/wp_hash_passwo... isn't in use anymore? Seems to indicate that the lowest common denominator is multiple runs of md5. I see that's pluggable, but I'm sure most WP sites don't bother installing an extra password hashing plugin, given a major part of WP's install base is one-click installs on shared hosting accounts by people that don't really understand what they're doing... (I say this from long experience, unfortunately.)

In any case, I'd love to find out it's not md5, that's just what the docs seemed to suggest.

I wouldn't completely trust the WordPress Codex for everything. It has a lot of missing/out of date information, and the function descriptions don't always jive with the actual code. That function is still in use and in this case it appears the information is only partially accurate. From what I've gleaned looking through the code, is that (in 3.5.1) WordPress uses phpass portable hashes. These hashes do ultimately rely on md5. The phpass class is initialized with an $iteration_count_log2 of 8, but the actual number of iterations of the md5 function itself is 8192 (based on my testing). While this current hashing is not perfect, it is better than an unsalted single pass of md5, or the only 8 passes as documented in the Codex.

I believe that phpass portable hashes were used for their cross system compatibility, as prior to 5.3 not all crypto functions for PHP were available on all platforms. With 5.3 all crypto functions are now built in, so it should be possible to use the non-portable hashes of phpass everywhere, but I believe that WP still targets 5.2. The version of phpass shipped with WordPress has support for non-portable hashes, and it would be possible to make a plugin that enables this, but as you've stated, it isn't likely that many WP sites have a hashing plugin installed.

Yeah, no, not really. "auto update requires web-write to your whole site" pretty much says it all, and the Wordpress plugin community isn't really known for it's security awareness.

According to the email (emphasis mine):

> This unauthorized access was made via third-party software installed on the Drupal.org server infrastructure, and was not the result of a vulnerability within the Drupal software itself.

There is ostensibly an argument to the effect of "well if they can't secure their infrastructure, maybe I can't trust them to secure the product code", but the Drupal infrastructure team is separate from the core maintainers and contributors. Your choice of CMS/CMF would've had precious little to do with this compromise.

Just my luck that a group of Drupal enthusiasts came upon my comment...

Oh really? http://techcrunch.com/2011/06/21/wordpress-org-possibly-hack...

Drupal is at first a CMS and WordPress is at first a blogsystem. You can not really compare them in the full spectrum of possible sites.

The difference there is becoming vanishingly small. There are _many many_ sites out there using WordPress as a straight CMS with no blog-like functionality.

I've used and deployed both - if I'd _ever_ been able to get one single client to understand the power of Drupal's taxonomies, I'd be developing and deploying Drupal sites everywhere. As it turns out for me, clients are _much_ happier with WordPress's backend/ecosystem/documentation.

I just get annoyed by the Drupal admin panel's absolutely boneheaded UI + missing features and slow performance - D7 was supposed to help with this, and we barely got a new admin theme. Wordpress isn't as sophisticated, but the UI is nicer and it has the basic functionality out of the box

You can save the patient by removing the brain, but this impedes potential.

Reply on news.ycombinator.com