If this is real, I'll be stunned if the auditor keeps his job.
Then again, after working at startups my whole career, maybe I'm just naive about how messed up the real world is.
Then again, after working at startups my whole career, maybe I'm just naive about how messed up the real world is.
The reason for this requirement to repeatedly pop up in bid documents is misapplication of a poorly drafted standard by people who are theoretically experts at enforcing it's normative intent.
I would love to throw in an additional anecdote here, but it would violate a confidence. Pretend that $500k had been allocated to produce a list of plain text passwords and a combination of bureaucratic inertia, competence issues, and internal politics made that requirement an unstoppable freight train. That gives you the flavor of it.
It's a lot of trouble to go through just because of the ignorance of a client, but it would be enough to check that magic box on the requirements list and be done with it, rather than having to go to all of the trouble of trying to educate the client and risk them going elsewhere.
(Although if you're using bcrypt to store passwords, which you should be, then apparently it's not so cheap.)
But.. 26 letters in the alphabet + ten digits = 36... and 36^n grows really, really fast. So maybe you'd need way more than 1k permutation checks.
Then when the user goes to change their password again, you have the plaintext of the current one, which unlocks the previous password, and so on back in the chain. You can then compare the new password with previous passwords for similarity.
First time: On his client, the user is asked to enter a password which is checked to comply with password rules. This password is salted and hashed before it is stored on a central machine.
n-th time: On his client, the user first enters password "n - 1" which is then salted and hashed and sent to the central machine for comparison. If hashes match, you may enter the new password "n" which is checked to comply with password rules.
Password "n - 1" is then encryted with password "n" and stored on the central machine. Password "n" is stored on the central machine in two forms:
1) Salted and hashed.
2) Encrypted with a public key.
Password "n" is encrypted with a public key and stored on a he central machine. Also, when this machine stores the new public key encrypted password, it deletes its own records of any previous public key encrypted password for the user in question.
Through recursion, all old passwords ("n - 2" ... "1") are decrypted and the new password is checked so that it does not match any previous password and that it is not a simple permutation of any of those.
If the user forgets his password, the administrator physically logs into a machine which is very tightly secured. This machine does not allow inbound connections of any kind over the network and can only make outbound connections to the central storage machine. Furthermore, the machine does not allow the usage of any removable media (CD, DVD, USB stick, etc.). The drives of the machine in strongly encrypted, as is the private key which is stored on this machine. In order to decrypt the disc of the machine, two passwords are needed. One of the passwords is held by the administrator, the other by the CEO of the company. The same goes for the private key which also has two passwords, both of which are needed to decrypt it. Thus, two people must be involved in order to decrypt the password for a user. Once the administrator and the CEO has logged into the machine, they connect to the central storage and retrieve the encrypted key for the user and then they open their shared private key and decrypt the password. After the user has been given his old password, he must create a new password following the procedure described above. That way, the administator and the CEO does not know the users new password (unless they conspire to do so, but that would be breach of trust).
I have come up with some astoundingly dumb ideas that were preceded by exactly the above thought.
However, there was some sheer brilliance, as well.
(Edit: so the match might be quick, but pre-comp of the hashes and storing them is not. how the hell would you store 1-10k hashes associated with a user anyway? :) )
So user feedback should be speedy enough.
Only once the password is deemed acceptable does the system need to pre-calculate the hashes for the next time the password is changed.
Of course, pre-calculating 10,000 bcrypt hashes might be too computationally expensive anyway, but the user wouldn't see a delay.
but I can't think of any other way of making 'similar to' work
Another problem is that to prevent the use of the very old passwords, you should keep the hashes of the modifications. To make the comparations fast, you have to use the same hash.
Now you have a list of #password-changes * #passwords-modifications, and if an intruder get them, it is possible to make a mini-rainbow attack.
You can give the user feedback on their new password choice after just one hash. The permutations would be stored from their last password set, so it would just be a simple lookup to determine whether a given new password is similar to an old one.
I'm in no way recommending this - it was just an amateur guess at a method to fulfill the requirements without exposing a massive security vulnerability.
I made the session tokens crazy long though, and have one for the user session and another for auth, and a sanity check within the auth token that will add any IP that attempts to forge the session token added to the ddos blacklist after 3 attempts. The auth token generation is slow as well - which is something that is often overlooked in webapps. I got a bit carried away in implementing all of it - but it was fun
The end result? A user starts with the password "Seekrit1" when they join the company, then the next month they change to "Seekrit2", "Seekrit3", and so on.
This is hardly what the security policy intended, so stopping that happening isn't actually a bad idea.
Obviously storing a complete password history in plaintext (which would even have to be online for the consultants' plan to work) is ridiculous, but pre-calculating a bunch of hashes of similar passwords every time a new password is set would certainly be feasible.
Changing passwords is usually a rare occurrence, but making this at least slightly computationally-expensive shouldn't be a problem. Of course, the more expensive the password-hashing algorithm, the fewer "similar" passwords you would reasonably be able to pre-calculate, which is an odd trade-off to have to make a call on.
One thing to be wary of, though: by pre-calculating and storing a bunch of hashes of similar strings, this opens you up to (what might be called) a related plaintext attack. I have no idea if existing password hashes are specifically designed to be resistant to this, but would guess not. (edit: I didn't think this sentence through correctly. Thanks, Joachim)
Which is almost as idiotic as storing a complete password history in plaintext, because it pretty much guarantees that passwords either (as you note) follow a simple pattern, or if that is made impossible, are written down in an easily accessible place.
bcrypt is just Blowfish(key-derived-from-salt-and-password, "OrpheanBeholderScryDoubt"), so it inherits this property.
Even if the datastore contained only password hashes (which is good), wouldn't it be (potentially multiple) orders of magnitude easier to brute-force password attempts in the event of a large number of potential permutations to be eliminated right off the bat?
Even though bruteforcing the stripped passwords wouldn't provide a full set of information, it might be a good starting point and quickly reduce the search or guessing space.
Disclaimer: I have no idea if this is a practical concern, just one that comes to mind when thinking about your suggestion.
This is where advances in hashing approaches might negate that problem for you, though - and you'd need a proper cryptographer to figure this out. If it was a plain MD5 or SHA of the normalized password, you're absolutely vulnerable to an attacker using a rainbow table of lowercase alphabet words figuring out the 'root' of the actual password; then they could use that to construct password candidates to hash and compare to the full password. But just adding a big block of salt to the normalized password though would take it out of the pure-lowercase searchspace and force you back to bruteforce searching of the (admittedly small) password space. However, multiround salted hashing strategies like bcrypt rule out rainbow tables and allow you to tune how expensive bruteforcing is, and even though the password space of the password kernel is limited, you may be able to turn bcrypt up high enough to make it unfeasible to bruteforce.... maybe?
I enjoy the juxtaposition of following the letter of the rule, violating the intent of the rule, but nonetheless honoring the spirit of the rule.
Unfortunately, it can't be extended to make sure that the new password isn't similar to the password from two generations back.
The only real solution is educating users. Comparing to the previous password (provided by the user when changing it) is close enough to the spec to be able to get away with it in most circumstances (possibly combined with saving a history of password hashes, so you can see if people don't repeatedly use the same password).
When a user changes his password, he enters the current password and a proposed new password. The system uses the current password to decrypt the list of previous passwords. It checks the proposed password against the list. If accepted, the old password is added to the list, and it's encrypted using the new password and stored.
Are there any threats against this scheme, or will it work?
Edit distance 3, even with "letter stays letter, digit stays digit, random additions and removals" will already give millions of variants, so this may be impractical, but impossible? No.
This is not a lone idiot in an otherwise sane industry. People like this can thrive and remain blissfully ignorant because the entire ecosystem around them is incompetent, from senior executive to junior trainee.
In our industry, entire companies operate in complete isolation, basing their practices on the knowledge of their most "senior" engineer who was there at the beginning, however clueless that engineer might be. Companies like this can continue to operate successfully for many years, moving from one client to the other, and you will find them in any niche of the software industry, including startups.
And being right in a court of law just means you win, it doesn't mean that you don't get your time, energy and money sapped so really, where's the benefit for the person exposing them?
Expose them within the company? Only works if your management supports you, and they are probably only doing a security audit so they can get a piece of paper with some initials on it. They just want it to go away, not to hire another auditor to start from scratch.
I'm afraid you might be. Having read the auditor's responses I recognized the tone instantly. Idiots like this do exist - they advance by throwing around lots of buzzwords and touting their "years of experience" which is often enough to bamboozle their equally incompetent superiors.
Try coming into Enterprise software :)