Prevent users registering with passwords from data breaches
jordanhall.co.uk
jordanhall.co.uk
1)not silently fail when I try a 64 character (or 32 character) password
2)not fail and say my password is "too short" when it is 32 characters and you have an unrevealed maximum password length of fewer characters than that.
3)just all-around quit failing when my password is totally fine, it's a quasi-random string of letters, numbers, and symbols and I'll never type it...
oh yeah
4)don't disable pasting in the password field
4b)IF you do, freaking let me see what I've typed. I promise to not enter the password in a place someone can shoulder surf. Trust me that's WAY down on the threat list for my life.
Then, and only then, if you really want, keep me from registering with a breached password. But you had better tell me that is what is going on.
What I find the most annoying is that developers enjoy creativity for password validation. Some requires special characters, but only from 2 or 3 allowed special characters (what good does it even do in term of entropy???). Some limit the number of times a character can appear in a password, and I understand that it is meant to prevent people using “aaabbbccc” as a password but it also makes long passwords impossible. Many have an artificially low length limit (why would they even do that)?
Do you want actual password security? We can do that. Otherwise, here's whatever nonsense is currently favoured by people who know nothing but set policy. Sometimes obeying policy _is_ negotiable but only for some users, so I could set the former configuration where I was able to negotiate and the latter for everybody else.
For example "passwords are case insensitive" or "no spaces" is likely to be in the latter because someone in customer support finds it easier. "Use any halfway decent salted, pessimised hash" may be in the former, whereas "Use this specific named hash that you'll have to implement by hand because it's not offered in the language and libraries used by the system" is more likely to be in the latter
I will say it's not necessarily "an exec that knows better". You're often playing a game of telephone with a third party such as a penetration test service, some government bureau or a "Being a CTO for Dummies" type text book.
When it's a game of telephone in a large organisation it's unlikely you'll be able to fix it, it only takes one person with political cover between you and the person who originally said something like "I don't know, 16 characters?" to ensure that's a hard requirement you're now being tested against.
My previous employer had systems that made it impossible for QA to get a "normal" customer experience, supposedly for "Network Security". A fraction of all pages loads at random are redirected to / on the correct site. All page loads. Image request, just logged in, made a post to a forum, anything might be turned into a request for the home page. Clearly it was a trivial config bug, and I even met people who knew which individual had made the configuration error, their name, where they worked, etc. But that person had political cover from VP level and so the bug was simply never fixed in the years I worked there. I've actually promised my entire ex-team lunch out if it's fixed while my retainer is still in place. I expect towards the end of the retainer I'll just go back to that city and take them out to commiserate, because the chances of it ever being fixed are so slim.
why would they even do that?
Pointy haired bosses often deliver solutions based on what sounds like an admirable position to take, or a point of view that makes them look good politically. One that sounds good in a rapid fire round table discussion, like a daily stand up, where the boss’ boss needs two sound bites from them because he’s VERY busy, and doesn’t have the time to think deeply before cocktails at lunch, and an afternoon of golf.
Q. Give me your update on the password business.
A. Very good sir, we’re making users pick strong passwords.
Q. What defines the strength of the password?
A. Well, we assign 10 requirements, to ensure the user picks something sufficiently random. More requirements is better than no requirements.
In the case of a password manager, if your computer is breached, then all passwords are breached. If your passwords are hosted encrypted on a website, then if that website is breached, the master password you send it will be visible to the attacker, and thus all passwords are breached.
In the case of using the same password everywhere, if one website is breached, then the password to all sites is breached.
Am I missing something?
You now only have to trust one website/program, rather than every website (and we have evidence that many major websites can't be trusted with our passwords).
Breaches online are very, very common, and I would expect any password used on every single website you use to get leaked quickly.
It really depends on your threat model. No system is perfect and we're mostly balancing security and convenience most of the time.
The same password everywhere is super convenient, but only one site needs to fail and it's all over. A cloud password manager with unique passwords protect from that. A local password manager eliminates the threat of them being compromised, but is less convenient if you have multiple devices. Storing your passwords offline on paper in a safe protects your from someone getting read access to your PC or outright stealing it, but is annoying and gets more inconvenient based on your passwords entropy.
I had used the one password on most websites years ago(I did not known better back then but I did used different password for my important email account), I think my credentials were leacked in 5 data leaks and hackers are still attempting to login with my old credentials in present. The only time this affected me when someone hacked and stole my Minecraft account but I got it back.
>if that [one] website is breached
>if one [any] website is breached
Not one of those things is like the others.
(edit: Actually, I've recalled at least a 7th service that has been compromised that HIBP doesn't list)
That's either 6 times my bank account password gets leaked... or 0 times if I use unique passwords. I strongly suspect there are more breaches that HIBP simply isn't aware of with my information in them. Additionally, I need to rotate either 600 (!) or merely 6 different passwords to re-secure my accounts. So, obviously, unique passwords are a big win here.
> If your passwords are hosted encrypted on a website, then if that website is breached, the master password you send it will be visible to the attacker, and thus all passwords are breached.
I use an offline password manager specifically because of this reason. Still, needing to pwn any of 5 targets to get my bank account (me, my bank, my password service, my email provider, or my phone provider) isn't much worse than needing to pwn any of 4 targets (me, my bank, my email provider, or my phone provider), and both are much better than needing to pwn any of 100+ targets, some of which have terrible security and are probably already pwned without even knowing it.
So, options for unique passwords:
Memorize 100+ strong unique passwords: I "can't" do this.
Memorize 100+ weak unique passwords: Too weak to brute force attacks for my tastes.
Use a memorizable unique password generation scheme: Not unique enough for my tastes.
Memorize a few very strong unique passwords and use a password manager. This does have the weakness that capturing my master passwords via keylogger will get you all my accounts, instead of just all the important ones that I regularly use - but since "all the important ones that I regularly use" includes my email, that's just as good as pwning all of my accounts in my book.
If they cannot get it right, how does that leave the rest of us?
Here's a python implementation using bloom filters which avoids storing the whole list (need to store ~1 gig), yet still gives you very good accuracy: https://gist.github.com/marcan/23e1ec416bf884dcd7f0e635ce5f2...
It's more what I think this should look like.
Too many potential pitfalls:
- If their service goes down, you may go down too, or you have the added complexity of gracefully handling the situation when they're down
- You have no control over their roadmap... which means you might suddenly get a bunch of non-value-adding but totally essential work dropped into your backlog, and perhaps at short notice, because of changes they make
- Perhaps they go out of business or, for whatever reason shut down their service: again, congratulations, you've just got a load of work you didn't bank on getting in the way of delivering your own roadmap
I think most of us have probably seen multiple examples of the above, if not at first hand, then posted on HN or elsewhere on the web.
I'm not an NIH kind of guy but I do tend to prefer libraries, or co-located installs for long-term dependencies. That way at least you can manage migrations and updates according to your own agenda rather than somebody else's.
In this way you get the best of all worlds; speed, highest degree of accuracy, and reduced dependency on a single external API.
For online attacks, an attacker can't even try the top 1000 passwords on for an account in any major website in reasonable time without triggering the alarm, as they all(?) have rate limiting (usually in the form of account lockdown after single-digit failed attempts).
For offline attacks, there first needs to be a breach. While they undoubtedly happen, they are very infrequent events. But once they happen, you should assume all passwords would be cracked very fast. Hackers can get their hands on a lot of computing power, and the brute-forcing attempts are not alphabetical, but rather clever how-humans-chose-passwords models. You're relatively safe because of the low frequency of breaches, not because a hacker trying trillions of passwords a second will be frustrated by your password choosing policy. I'm sure most of the passwords from the breaches would be attempted anyway.
Credential stuffing[0] is the real issue. If there was an API to test that a user isn't using this password on other websites, that would be very useful.
Before that, attackers would just wait for new usernames to appear on the scoreboard/chat and check them against password dumps. The easy come, easy go nature of bitcoin made it particularly lucrative.
Password reuse is a massive issue. At a glance, one might wonder why most sites need to care so much since they don't deal with money. Who cares about a forum like HN? But consider that it's easier to audit/impede new accounts with anti-spam measures, so there's value in taking over old accounts. And you don't want people with moderation tools getting attacked either. Ideally, it's nice to be able to trust an account with 1,000 posts more than one with 0 posts, but that evaporates when accounts are easily stolen.
Aside, how do you implement account-locking without making it trivial for users to DoS each other that way?
By adding an (increasing) time delay after each failed attempt. However many sites, banking in particular, just lock your account and you have to call them on the phone to reopen it. Totally open for massive DoS but the world still stands.
Notice that banking websites can get away with it, yet surely you weren't only talking about banking when you said "online attacks".
Because it's trivial to implement decent ratelimiter which stops attackers, but still allows normal users to login.
But if care about security, you might as well solve the problem properly instead of wasting your time implementing "your password must have a capital letter and a number" snake-oil features which only annoy your users.
It's also not a trivial issue. It's cheaper and cheaper to attack a website with unlimited IP addresses. Which dimension are you going to rate-limit?
There is no lock that cannot be lockpicked. The only difference between a good lock and a bad lock is amount of time it takes to lockpick it.
And captcha only makes brute-force somewhat more expensive. You likely have to use a 3rd party captcha service (like Recaptcha) which incurs network volume amplification since you need a req/res to Google just to render GET /login. It also shows how hard of a problem captcha is that you can't just roll it yourself.
You're still not addressing the problem since IP addresses are so cheap. I only need 1,000 IP addresses to try the top 1,000 passwords in parallel no matter what your rate-limit scheme is unless you plan on letting me lock the authentic user out of their own account.
You've wandered from your original claim that bruteforce doesn't even happen these days, and I'm certainly not saying we are helpless against it. In the end, I'd simply suggest that this problem is harder than you originally gave it credit for, and maybe that's something we can agree on.
You want simple concrete example of how to stop all that "unlimited amount of ip addresses with captcha solver service" stuff? No more than 20 attempts per day with ip different from the ip of the last successful login. Here it is, you just solved the "bruteforce problem". That's all you have to do. Other things are just quality of life improvements.
This is empirically a practical attack: attackers successfully executed a common password brute force attack against GitHub in late 2013 by using a botnet with 40,000 distinct remote addresses:
If it cares a little bit then you have to characterize exactly how bad it would be if the account was compromised. The right solution might be to just accept any non-blank password that's less than 32 chars or something.
We should honestly move to the world where typical consumers are using password managers that generate passwords randomly. I think it is pretty reasonable to expect the typical consumer to install and use a password manager; I think it's pretty unreasonable to expect the typical consumer to generate memorable, strong, and unique passwords for each website and store them in their head.
"It has to be new" is about as elegant as you can get for password rules. You can use any secure method you want. Whatever characters you want, just don't do it wrong. If you want it to be easier, press the button.
Even with a one-off account, a complex password you don't bother to remember is better than a weak password, because the service won't have to deal with tons of misused accounts.
I say better for the service theoritically because I probably won't sign up (less of an issue since I moved to LastPass, but still annoying because so many sites don't work with autofill.)
Personally I use 1password, but there are loads of other ones that don't have a list of security breaches on their wikipedia page.
Once the password is broken, the account will immediately be placed in suspense until the owner creates a new password. Which, of course, immediately gets fed back into the machine and life goes on.
This would eliminate password rules. If you want to create a password which consists of, say, 100 consecutive zeros, go for it. But you might only get to use it for a fraction of a second if the network can break it quickly.
- It's already fairly straightforward to predict how easy a password will be to crack e.g. if you're going to bruteforce dictionary passwords or those with predictable combinations of characters, why not just include those in your password blacklist? - Randomly suspending users' accounts and telling them to change their passwords is going to annoy them (especially if it happens repeatedly) and consume support resources - It's overkill unless the service is security critical - How easy a password is to crack is partly a function of the hash algorithm and salting the entity uses so it may not be the user's fault their password gets broken
You can just do a case-insensitive match against this file that I compiled a while back: https://github.com/robsheldon/bad-passwords-index
It includes the most commonly reused passwords according to in-the-wild breaches.
I'm a bit embarrassed to see that it's been 2 years since the last update. I was thinking recently about updating this again. I think I'll do that.
Seems like eventualy you'll end up driving away everyone who isn't tech savvy/doesn't use a password manager/randomly generated password, which seems like something that'll significantly limit your site or app's audience.
I get the logic behind it, and it's a neat idea on a security level, but it seems like a guaranteed way to drive your userbase to your competitors by making it annoying to sign up.
This type of question should always be asked with an "XKCD What If?" or "Back of the envelope" calculation attached so that when responding people can respond to your numbers rather than a vague intuition like "large percentage of standard passwords".
This way when doing the calculation you sometimes go "Oh, I see now" and then you don't have to post the question at all because you learned something on your own just like if you are wondering what a "password" is and then check a dictionary you don't have to ask "What's a password?"
What's a "significantly large percentage?" Let's suppose it's 10%. If 10% of these "standard passwords" are disallowed, and a user is only willing to try twice before giving up then that means 1% of users picking random "standard passwords" can't sign up, which we could argue is a noticeable problem for your site.
Next we need to ask ourselves what's a "standard password"? Do we just mean "a password from the top 100 most common passwords list?". If so game over, we've decided by policy to have users with bad passwords that will be guessed, their accounts will constantly get broken into, and I guess now we need to figure out why our service is "compelling" enough that you want to pay for it anyway despite the constant break-ins.
So let's suppose we have a broader definition of "standard password". Maybe it's 8 characters chosen from A-Z. That sounds like a "standard" password my mother would choose. That's over 200 billion "standard passwords". Troy's service blacklists, wait for it... 500 million passwords.
Let's guesstimate that the list grows by half that amount every single year - meaning 250 million new unique passwords are revealed by idiots every single year. We should hit that concerning 10% rate no sooner than... 78 years from now. If in that time we cannot come up with anything better than "try to memorise a unique password eight letters long" then that's a far more serious technical failure.
The real problem is that people think they must be the only person in the entire world to have chosen "iluvlucy". That's a "standard password" by our definition above, but it's also ludicrously obvious. Pwned Passwords lets us distinguish "iluvlucy" from "xlvghydm" not by some crappy heuristic that would also catch other things but by the fact that lots of people already used it and had the password revealed.
Microsoft research has an interesting paper on it. Are there more like this out there? https://www.microsoft.com/en-us/research/wp-content/uploads/...
Hints welcome!
Many users have a "universal weak password" for sites that don't really matter, now you will be forcing them to jump through hoops just because so.
And I think forcing people who are in a position to use unique passwords easily, but too lazy to do so, to get around to using unique passwords is a good thing to do. I include myself in this category - I was sloppy at password hygiene until very recently and I should have gotten on it a long time ago.
(Note that I'm not endorsing password schemes because they're very vulnerable to targeted attacks, but they are popular and arguably easier than password managers and they do technically count as letting you use a unique password on each site - if you use one, the HIBP API will not block you from logging in to any sites other than the one that got breached.)
Wouldn’t a better solution be to increase password requirements until users are forces to generate one using a password manager? If you can memorize a password it’s probably not secure.
So, to access your bank now crackers just need to get in to your email (which may be true anyway, of course 2FA helps in both cases).
The benefits of sign-in links are: a) you don't have to remember gazilion of passwords b) you don't have to use password managers c) you can setup a really strong passphrase for your email and actually remember it because it's the only one you have to. And of course set up strong MFA or whatever d) you don't have to setup MFAs on services and giving them more personal info(phone number) than they really need e) for me as a developer it's also easier to implement actually
User: Set my password to 'monkey'
Website: Sorry that's a common password
User: OK, set my password to 'monkey1'
Website: Sorry that's a common password
User: What?! OK, set my password to 'monkey123'
Website: Sorry that's a common password
User: Grr! Set my password to 'monkey123fuckyou!!'
Website: Sorry that's a common password
User: Screw this, I'll just sign up to your competitor's website instead
I wouldn’t recommend it, but you could try a slightly scarier explanation. (“Sorry, hackers have cracked this password on other sites. If you use this password elsewhere, you should consider immediately changing it.”)
The logical conclusion of a password checking system like this is that this password:
ZBjHWJd$8XbJhY7LQvkmARBW)p7xgiDzDw}iMLLw
In an offline brute forcing scenario, either the passwords have been hashed with a strong key derivation function and a randomized salt or they haven't. If they have, it doesn't matter if the password is in that list. If they haven't, you're most likely screwed either way, because attackers can get up to 1 trillion password attempts per second in real world cracking setups now.
Extrapolating further, this "policy" would disallow people from choosing passwords below a certain length, because it's theoretically plausible someone has published the set of all possible strings fewer than n characters long.
Yes, that seems like a good password policy. A list of possible alphanumeric strings that is actually reasonable to physically publish (i.e., not 20 trillion) is a list of extremely short alphanumeric strings. 5 alphanumeric characters is 380 million possible strings. 6 is about 2 billion. You should absolutely ban passwords that are 6 characters or shorter!
In fact, I would go so far as to say that the questions of "Is this password too short because someone could brute-force all the possibilities, even if we're using a good password hash and a previously-unknown salt" and "Can someone physically enumerate all passwords of this size and put them on Pastebin or otherwise get them in the HIBP database" are equivalent.
And this is aside from the fact that you won't even rip through those 20 trillion in an online brute force attempt. Eventually you are only constrained by computational resources. If the passwords are cryptographically secured, it's fine if any one of them is published on the internet if you cannot associate it with any given user. If they're not cryptographically secured, this won't meaningfully reduce your already poor security anyway.
My principle is that you should not let people sign up with breached passwords at all - don't make judgment calls about which breaches matter, and whether you think it's the same user or not, or the password is strong enough or not. Just ban the passwords. (Remember that no actual data breach contains 20 trillion passwords.)
The world has decided that making me go through a Google CAPTCHA when registering a new account - a process that takes me several seconds of active mental effort - is fine. If the check takes even 1 second of server time, is that noticeable?
There are MUCH smarter ways to implement this than grepping a list line by line. I mean seriously who the hell would do that... If your bio isn't complete BS then surely you understand that proposed implementation is BS.
That being said I'm actually going to concede this argument, because on further investigation Troy Hunt provides the entire copy of his database freely for local lookups. Once you hash the user password to match the database hashes you can introduce the optimizations you mentioned, and in any case it shouldn't introduce intolerable latency to do that lookup locally.
Blocking a list of 20 trillion passwords is probably overkill if you have a slow hash. But with a fast hash it's the difference between "impossible" and "less than one GPU-week".
And if it's easy to block, you might as well do it. There's no upside to letting people use already-posted 20 character strings.
If you publish one trillion passwords from a large space then each one of them gets a probability boost (of approximately 1/1 trillion), though not enough to ban them, especially if they are not actually associated with accounts.
The danger with using a rare but breached password is that there is actually quite a high chance that it was breached from your account elsewhere.
I do agree that if the entire space of possible passwords is only 20 trillion, that doesn't change the probabilities. But there are over 20 trillion eight-character alphanumeric passwords. So, I would actually say you should ban them all, because you should insist your passwords are at least eight characters long. :-)
Edit: yes, agree, in practice the probability boost is not very much. I'm just saying you may as well ban them on the assumption that the HIBP API will do so at its current level of performance. (20 trillion is a ridiculous number, because it's much larger than any possible breach and yet much smaller than any meaningful password space, so any arguments about it are going to be inherently silly in some fashion. My current silly assumption is that the HIBP API is capable of ingesting 20 trillion breached passwords with no performance hit.)
If not then it really doesn't matter that they got published. They're useless to hackers without knowing what email to type in. The attack model is that the attacker actually has to log into a website and you don't get 20 trillion attempts.
> The attack model is that the attacker actually has to log into a website
Not to find the password. If it was then nobody would get upset about plaintext password storage.
But anyway if you pick a password from a list of 20 trillion where the offline attacker knows the list, it doesn't actually help them much because a single selection from 20 trillion options has 44 bits of entropy.
Passwords that users choose typically have less entropy than that afaik
No because they each still have a very low probability.
You have to be Bayesian about this: a list of one trillion passwords that have no further distinguishing information about each one of them cannot be assigned a probability of > 1/(1 trillion)
In a data breach, a given password appears next to a particular username or email, which means it has a very high probability of being the password for that account.
Remember the "Debian weak keys"? What's weak about those particular keys?
Nothing. Nothing whatsoever. Those keys aren't special in any way. Except, Debian shipped releases that always picked one of these key pairs. So anyone with a mind to can go find the list of private keys that corresponds to these particular public keys and thus we don't let you use those public keys any more.
(You can go try this if you don't believe me, submit a CSR to your preferred public CA asking for a certificate for one of the Debian Weak Keys, it will be rejected and there may or may not be an explanation attached saying your keys are crap and to get new ones)
Whole swathes of keys are blacklisted. ROCA is another example, somebody took one mathematical short-cut too many in their optimised design for RSA key generation, and so the resulting keys all have this very obvious structure that's exploitable (not easily, but enough that a sovereign entity could definitely break them). So we just blacklisted all those keys.
If you pick truly random keys you'll never notice this in a lifetime because of statistics, it's just some code on the issuer's systems that you never need to care about.
edit: I did try logging into your account with the password you posted. :P
Specifically if it appears n times in the HIBP database you should assign at least roughly 1/n probability that the user is reusing it.
So if you assign disutility -V to letting a user have a known username + password combo and utility U to letting a user sign up with a known password but unknown username, the utility is (n-1)/n×U - 1/n×V
Reasonable values of U and V for a given site will be different depending on the application, but for online banking -V would be maybe -20 and U might be negative as well. You wanna bank with a public password lol? For something like gmail or Facebook it would be the same story.
On the other hand if the password is quite weak then it's vulnerable to credential stuffing. If it appears, say, 10,000 times in the HIBP database then most likely it's as good as public whether or not the user account name is known.
Maybe there's a sweet spot around 50 instances where you can't really credential stuff it, and you also aren't that sure that it's a reuse.
In terms of usability you could tell the user to change it up a bit, add some words.
For example, r0bbiewilliams appears 5 times in the database. luvrobbiewilliams appears 0 times AND IS PROBABLY EASIER TO REMEMBER!
You can almost always get away from a breached password by adding a small amount of text.
I'm not talking about scenarios where you can associate the password with a specific user.
And HIBP will tell you how many times a given password appears, but not which account it appears with. See:
yes, exactly.
If it appears 2-3 times, then there is still a significant chance that that user is the source of the password getting into the HIBP database.
And if that user is the source, then the bad guys most likely know that user's email and password, and their account is wide open.
1) Extremely weak ones that lots of people use (e.g. 'password1') 2) Somewhat unique ones (their pet's name and birthday) 3) Truly strong ones (random, long strings)
I don't want users on my site using type 1 passwords at all. If a password is really type 3, the odds say that no user will ever try to use it again, so there's no collateral damage in blocking it. The person signing up with a type 2 is almost certainly the same user whose credentials are in the breach. I don't want them to reuse that password on my site because it makes their account vulnerable to credential stuffing.
Passwords already suck. Please don't make them suck even more.