It's still bad, I'm just saying the conversation about what hash algo to use didn't happen yesterday.
Unfortunately... that argument wasn't wrong.
The way I would have implemented it, but would be keen to know how secure it is, is that you start with the md5 of the password (md5(password)). You then bcrypt or scrypt that md5 (bcrypt(md5(password))) and replace the md5 in your database with the bcrypt hash.
When a user logs in, all you need to do is to calculate the md5 first then check that md5 against the bcrypt hash you have stored.
I am not a crypto expert but intuitively it doesn't look like I would have weakened the security that way. You can't really attack bcrypt(md5(password)) much more than bcrypt(password). Can you?
I didn't do the test, but I'd expect that there wouldn't be more than a handful of collisions for the md5 of the 100m most common passwords.
[edit] I actually I just did the test on this 10m password list and no collision
https://xato.net/today-i-am-releasing-ten-million-passwords-...
Yes, collisions are easier than preimages, but they still shouldn't occur by chance in real applications!
MD5 has a 128-bit output so collisions that occur by chance should require about 2⁶⁴ inputs (18 exa-inputs). Surely your database didn't contain over 2⁶⁴ different movie records.
Could you take a look at what you were doing again? Your description doesn't really make sense mathematically.
The security problem with MD5 isn't collisions.
Not "many different" using the normal constraints of text/numbers/typographical-marks and with maximum password lengths of 32 or so (I'll bet Yahoo's was shorter than that in 2013).
Are there any MD5 collisions in [:graph:]{,32} ?
0e306561559aa787d00bc6f70bbdfe3404cf03659e70 4f8534c00ffb659c4c8740cc942feb2da115a3f4155c bb8607497386656d7d1f34a42059d78f5a8dd1ef
(Yes, susceptibility to collisions was recognized as a problem with MD5 leading to a reason not to use it, but the collisions in question were constructed, not encountered accidentally. There isn't any evidence to date that the probability of a collision given two randomly chosen inputs is higher than the expected 1/2¹²⁸. You could test this yourself by hashing 2⁴⁰ random strings under MD5: you won't see a collision among the outputs!)
> and do a "we sent you a reset email" for anyone that's trying to log in but has no <stronghash> password hash.
Yahoo is an email provider so many of these users won't have an external provider to refer to.
The other way is to add a new empty column for bcrypt. The next time the user logs in, you save the bcrypt hash and you remove the MD5 hash.
Over time, the active users will be migrated to the new scheme. The only issue is the abandoned accounts, they'll keep the old weak scheme.
For a website like Yahoo with billions of abandoned accounts, that's a serious drawback ^^
Usually you do this by decorating the bcrypt(md5(p)) entries in some way so you can recognize which ones are tested with bcrypt() vs bcrypt(md5()).
Wouldn't engineers at such a big corp whistle-blow such incompetent decision making?
Apparently [1] they had a $1.37B net income in 2013. Given using bcrypt with a Blowfish hash and salting was pretty much a de facto standard by that point (I think that's what Wordpress were doing, hardly revolutionary security work) it seems the relative cost for Yahoo was approximately zero.
All I can imagine is that those in control were asked to leave the system open for government snooping? Why else would engineers working there not [anonymously] bring this to press attention - "hey, Yahoo security amounts to a piece of sticky tape holding a bank-vault shut".
- - -
[1] http://www.marketwatch.com/investing/stock/yhoo/financials#
"We do not store passwords as a plain text in database. We have functionality which encrypts and decrypts passwords. We have only ecnrypted passwords in the database.
Almost all other servers use one-way encryption. In this case, passwords cannot be decrypted from hashing."
Again, this is a Bay Area based shop. For code written in 2016.
I was shocked to receive this, but it (among other things) leads me to suspect that there are lot of people out there, in positions of power, who aren't just ignorant, but who actively cling to password-storage anti-patterns.
I'm at a loss for how to fix this.
That's insane...
I'm not really a software developer but I really can't imagine it being a huge change. Instead of md5(pass) you could probably just change that to secure_hash(md5(pass), salt), add another column in the database for the salt, and rehash all the passwords. Customers wouldn't notice. Rehashing the databases would take a while, but otherwise that's really not a huge amount of work.
I did ask about the hash of hash thing some time ago and ptacek claimed that's a reasonable thing to do.
Oh no, only 128 bits. The NSA will be able to brute force one of those passwords in 80 years.
The typical way around this is to create your new destination column (e.g. sha256 with salt), and progressively have applications reference this column rather than the MD5 unsalted column.
It's a huge amount of work, and if the applications were made in 1990's, the code is likely legacy. If Yahoo are doing regular code security reviews, this will likely have been put in the pile of "we need to fix, but it's too costly to do".
Which begs the question, can legacy code survive in an international network?
A large organisation will implement layered security (otherwise known as layers of the onion) to prevent this type of attack. This means; more secure passwords to access the password database, fewer people with access, rotation of access passwords, auditing of backup storage and encryption, etc etc. Clearly Yahoo's layers of security were all broken to allow this type of theft.
Really? Moving from doing md5(password) to bcrypt(password,salt)? I see organisations make things hard and legacy code-base, yadda, yadda but surely if Yahoo couldn't do this then they couldn't manage scratching their own butt; it really seems like quite a small change in the scheme of things. Like one senior engineer, one afternoon of work (then testing, etc., OK, sure) ... ?
I'm going to go out on a limb and guess you've never worked as a software engineer in a large organisation.
Given MD5 hashes are currently stored, how do you propose user's password get converted to SHA256/512? Should Yahoo brute force the passwords, and then store them in the new algorithm? Or should they wait for the user to log on, verify their password, and store it in the new hash algorithm (given some users rarely log on, this could take over 12 months to complete 80% of users).
Even if it never completes (abandoned accounts), it would still have saved most active accounts from being breached.
I was just replying to the comment it could be completed in an afternoon.
On the storing of hashes though the standard protocol has been to pass the hash in as if it were a password.
(I've never had to do this myself, so these are just the most obvious options I came up with. Possibly there are others.)
The password "foo" may encrypt to the hash "12345". If an attacker were to discover that the hash is "12345", they would look for a password that hashes to "12345", which could, hypothetically, be the password "bar". They don't know the original password "foo", they've simply discovered an alternative, which happens to match the algorithm enough to unlock access.
In general, rainbow tables are used for identifying and attacking common passwords, but that doesn't mean that the algorithm is insecure.
Insecure algorithms can be attacked through collisions, which don't necessarily give you the original password, they just provide an alternative password which is accepted by the algorithm. The distinction matters when it comes to password reuse, because if Site A uses MD5, but Site B uses sha512, finding a collision that grants access on Site A doesn't necessarily give you a password that will grant access on Site B.
Somewhere in the organization, a product team is going to throw a fit about usability and churn over the decision to reset user passwords en masse, or to force users to change them when they first log in. This isn't a slight against product managers, but one of the clearest indications of a company's overall security culture "health" is how the security, engineering and product teams choose to compromise and "pick their battles." Risk accepting vulnerabilities has a legitimate place when you have to balance product development and usability, but so does pushing back on egregious issues.
I don't have privileged insight into Yahoo's organization, but in this case it's pretty clear the security team should have either been more diligent in conveying the ramifications or less kneecapped by the surrounding org units, depending on the circumstance. More importantly, Yahoo should have "migrated" their passwords in the manner a parallel comment explains in this thread. This is what Facebook and other companies did after maturing their security programs (see "Facebook Onion" on how Facebook transitioned away from MD5).
Also good to note - there is evidence Yahoo's security culture improved over the years. The decision to go with MD5 almost certainly happened in the 90s, and when Tumblr suffered a breach all users were forced to reset their passwords. The capability and awareness was clearly there.
You can only rehash if you have the plaintext password
Drupal 7 used such a technique for upgrades from Drupal 6, migrating from MD5 to a salted sha512 hash, but it's not an uncommon technique.
The old passwords are stored as MD5 hashes in the databases. The MD5 hash is processed through the same techniques as new passwords: a salt and the new sha512 hash. Provide a way to identify whether the origin was a password, or an MD5 hash.
Either way, you end up with a hash. You can identify whether the origin was a password, or an MD5 hash, but you can neither determine the origin MD5 hash, nor the origin password, as the new hash is secure. So even if the original MD5 hash was insecure, the new hash is secure.
When someone attempts to login, you still need to determine which password-validation to use: hash = sha512(salt + password), or hash = sha512(salt + MD5(password)), but the security level is the same.
Passing the password through MD5 reduces the complexity to 128 bits, you can't get that back.
So the security level is not the same, though it may be resistant to some attacks on MD5.
And it's probably not important for most people, since there are less than 2^56 eight character ASCII passwords.
> "Passing the password through MD5 reduces the complexity to 128 bits, you can't get that back."
Attacks against MD5 are based around knowing the original MD5 hash. In this instance, the original MD5 hash is unknown, so there is no mathematical shortcut to finding a collision.
The attacker needs a password with a specific hash, and the best reported attack for that is around 2^128.
Personally, I'm willing to chance that my password will be discovered via a brute-force attack within the next 0.65 billion billion years [1]
[1] http://bitcoin.stackexchange.com/questions/2847/how-long-wou...
A new preimage attack could be discovered - or might already have been, secretly.
No, this is not the problem with MD5. You are not going to find two user-memorizeable-and-typeable passwords with an MD5 collision.
If you are bringing a password with more than 128 bits of complexity to the party, any password storage scheme better than plaintext will have your password safe.
Collisions are a problem for digital signatures, not for passwords.
But some people do want and use more than 2^128 bit passwords, for whatever reason, and an MD5 intermediate stage limits that.
If I had a nickle for every time I've heard this statement then I'd have enough to comfortably retire.
Yes, in theory, changing a column in a database (which in this case, happens to be a password) seems simple, but in practice, it's not.
It's easy as hell. Even PHP, so often flamed for "bad security" these days supports EASY functions for this (and polyfills are available, if you're running PHP < 5.5, which you should't do anyway):
- password_hash, which creates a salted hash (the returned value consists of a type/strength spec, the hash, and the salt)
- password_verify, which verifies a password with a hash in a timing-safe manner
- password_needs_rehash, which tells you if you should update the hash in the database
password_hash and password_needs_rehash take a parameter for the hash function (currently only bcrypt is supported, quite likely to keep people from using md5/sha1), and for the cost (the amount of hash function calls).
I believe any reasonable programming language these days has such functions.
What I am NOT so sure about is how the various LDAP server implementations, which many people use for SSO and "normal" account management (because it's easier to connect a new software to LDAP than to migrate existing user db's into LDAP), handle password storage. I mean, having an LDAP server for the credentials prevents any form of password leakage, but in case someone breaches both servers/the LDAP daemon is running on the same host as the webserver?
If anything goes wrong with the password update, users get angry, lose faith in the services, stress, a few people get fired maybe, etc etc. On the other hand, letting it stay old and crappy just everything stays just peachy, and nobody is the wiser that the entire system is a house of cards. Until the day someone hacks the database of course... which happened so its "now" a problem.
They're not going to begin to take security seriously even after this incident. They'll do what they need to right now but there's no auditing and their users don't normally care about this sort of thing, therefore the management won't care either.
Rehashing can be safely implemented as long as the auth. process can handle both md5 and some composite hash [i.e. shash(md5(pwd))]
It's really a trivial operation.
In reality, they're constantly under pressure to develop new features, fix reported bugs, move on to the next project, keep the site from falling over, etc etc.
And the ones who choose NOT to work hard aren't sitting around reviewing old code either.
Still neglectful, but I sincerely doubt it was just a recent engineer's bad decision-making.
You assume a formal decision was made? I think a manager just went "make them secure" and history was made. That's how it usually seems to happen if it's not a user-facing thing.