Show HN: 5+ Billion Passwords in Order of Most Popular | Hacker News Reader

Show HN: 5+ Billion Passwords in Order of Most Popular

github.com

184 pointsberzerk052 comments

This list is immediately useful for validating user-created new passwords. Just stop with the bizarre rules about having uppercase, lowercase, symbols, numbers, length, etc. Instead require a string not in the top 10K (or 100K, or 1M) most popular.

Since the list is only rarely updated, this seems a perfect application Botelho's for minimal perfect hashing algorithm[1]. At about 8 bits per item storage and constant lookup it would be quite practical to use the top 32 Million list (appeared at least 10 times).

[1] http://cmph.sourceforge.net/papers/tr06.pdf

Speaking of rarely updated - version two will come out mid-July at the latest.

I'm also going to attempt to attribute the sources to given countries or at least language families. The list as it stands has a heavily US-centric bias.

I plan to grep through it and pull out things like Cyrillic or Arabic characters, and then repeat the process using only those characters. That's not to say that everyone who uses Arabic or Cyrillic characters is guaranteed to use them, but it's more specific.

If "password" is #2 in the world, perhaps пароль (password) will be the #2 most popular password with Russian Cyrillic Characters

I wonder about how to package your data as developer tools.

There should be libraries in any language used to develop systems which validate user-entered passwords.

Dropbox has their zxcvbn method which incorporates a list of 30k of the most common passwords

https://github.com/dropbox/zxcvbn

It looks like they gzip the file over the wire and then work with it uncompressed. That's probably a reasonable choice given they use only 30,000 passwords.

"Your password must be between 12 to 46 characters long and must include at least one number, upper case character, special character, kanji character, rune and quadratic equation"

"It also needs to be changed every 60 days and cannot be the same as one previously used in the past year."

(Promptly changes around variables in quadratic equation for the umpteenth time, leaves the rest of the password the same.)

I used to just tack on Aa$0 to the end of all my (already good) passwords to satisfy these idiotic rules. Until one website decided that '0' was not a number, and then I had to change everything to Aa$1.

Until a website thinks a 1 is an lower case L, and you have the same problem all over again.

I tend to revolt against this kind of bullshit by creating terrifyingly primitive, pathetic passwords, and flagrantly sharing what my password is, in front of relevant admins, while complaining loudly about the concept of "character diversity," as a way of fomenting popular rebellion.

When I'm met with periodic expiration, I employ constant direct password assistance as a means of unpopular two-factor authentication, to piss people off, and stir the pot.

I then fail my password reset as many times as necessary to recover my preferred password for that system, to enforce to freedom of password selection.

I control my password. Me.

Not you. Me.

Im glad Im not the only one that does this.

But it cannot include ",',$,;,<,space,etc.. because or engineers are still not 100% confident in our input sanitization.

You'd be surprised how many websites won't accept SELECT or DROP as valid usernames.

And you can be like HSBC which along with not accepting special characters, also saves and validates your password only in lowercase

And Citibank internet banking passwords are case-insensitive. (Atleast I can vouch for Citibank India)

Facebook passwords are also case-interchangable. i.e. PasswoRd5 is same aa pASSWOrD5 both will work. Idea is if Caps lock is left ON by mistake.

This is precisely why I don't use ubNUTu.

Note that this password list is valid rather for English speaking world.

you should package this up as a bloom filter and simple js routine web developers could use to do client-side checks to validate passwords.

edit: on 2nd thought looks like a bloom filter for 5B entries at p=0.01 would be ~5GB, so not exactly convenient

The largest list is 20GB, but it's not the only one.

Popularity was based on how many they appeared in files that had all duplicates removed (in reference to themselves)

The smallest file had passwords that appeared 75+ times, and the largest file had passwords that appeared 2+ times.

The top 195 Thousand (which appeared 25+ times in analysis) clocks in at 803kb as a text file with nothing but the passwords themselves

Password validation is stupid and annoying. At best this should be used to display a warning that can be ignored.

Somehow I wonder if life is significantly different for people named Daniel.

Or Alexander or Victoria. It's weird that these first names appear in the Top196-probable.txt file.

Having issues with the Seedbox, torrents are down temporarily.

The main page contains links to Mega.NZ alternative downloads. Will be fixed shortly, apologies for the inconvenience.

Is there really any reason to multiply the downloads by 4 just to have different compression methods? It seems to me like that just needlessly dilutes the seed swarm and wastes space, since pretty much any modern archive reader can unpack all the provided formats. Sure, specialized command-line utils can't, but if you're using one of those then you probably know which one to use for a given format.

You know, you're not the first person to bring that up - and I am having seed problems.

Perhaps next rev I'll only include the .7z (smallest) and greppable formats.

Have you considered using IPFS for distribution?

Hadn't heard of that before, I'll look into it

Torrents are back! Mostly.

Two out of the 12 aren't, but every wordlist can be downloaded via torrent in at least 3 compressed formats, including .7z

I don't understand: Why do you assume that password checkers keep their lists in alphabetically sorted form rather than just to load the whole thing into a db table with an index on it?

As I was looking around for the files to make this project, on SecLists, Weakpass, and Hashes.org, most of the files were in alphabetical order. This was especially true for the larger files.

Yes, but it doesn't matter what order they are in if you put them in a DB. I don't know anybody that would do a sequential scan over a file in production.

All you've done is shuffled around the most frequent cases to the beginning, which is great if that's what you are looking for, but now something else occupies the last slot and that case is just as bad as before.

The real solution is to see these files as input to an indexed table so that you can get to the entry you want in log(n) time.

I don't know anybody that would do a sequential scan over a file in production

Well, that's pretty much how one would try to crack a password using a wordlist.

EDIT: If the goal is to crack a bunch of properly hashed (PBKDF2, scrypt, etc.) and salted passwords then a lookup table is not very practical.

If you're trying to crack 'a password' you're doing it wrong. Alternatively, you're doing something illegal.

That's not why these lists are made public (though I can appreciate the fact that they are 'dual use').

PP is talking about a lookup, you are talking about cracking. Different things. For lookup of a string match make an index.

Not to sound negative, but is this something we should really be exposing? It feels like the only ones who gain from this are hackers and password crackers, no?

To quote a certain Alfred Charles Hobbs: "Rogues are very keen in their profession, and know already much more than we can teach them."

Also see: Security through Obscurity.

This doesn't have anything to do with 'security through obscurity'.

It might not be. When someone raises concerns around disclosing weak passwords or weakness in crypto, or security holes in general [0], I kind of like to think they are trying to think abt the problem in the wrong way. STO is a good framework, I think, to help people steer in the right direction, and hence the 'also see' label.

[0] Google's notorious Project Zero, for instance.

STO is a good framework

It's a fairly specific term so trotting it out for everything makes it pointless and obscures its actual meaning. Every time someone misunderstands something about information security (or someone thinks they do), someone else comes along and says 'Security through obscurity!' or 'Trusting trust!' or something about source code. It turns these things from terminology and concepts into magical-sounding incantations that really just mean 'I disagree/don't like this'.

This is analogous to the more familiar overuse of 'ad hominem' and 'straw man' and so on. It's not a good thing.

I get your gripe with the overuse. I now agree with the point you are trying to make. Thx.

Of course it has to do with security through obscurity... Passwords themselves are only security through obscurity.

No, they aren't. Not knowing something is not 'security through obscurity'. If it was, any encryption at all is 'security through obscurity' because, hey, you don't know the plaintext. That isn't what the term means at all.

Audits. Education. Change.

It's not like bad guys don't already have wordlists, or that they are new in any way. Having good ones available, in the open, for everyone to use, provides a net benefit in the long run imo.

This is similar to my train of thought.

Year after year people still use the same passwords that were the most popular for the year before. I think we should make it as obvious as possible that it is worth your time to make secure passwords.

I think people know that they shouldn't use "password" as a password, but it just needs to be made very clear through safe means (such as publishing wordlists that don't have any associated user data) that it leaves you very vulnerable. Many people simply haven't stopped to wonder how secure their password is.

My goal with this project is to flip the status of the passwords on the list - and make them the LEAST likely to be used.

It's already exposed. That's the point. The adversary has this intel. It would be irresponsible not to study and publish it, thereby reducing their informational edge.

Like others have mentioned, the criminal hackers are already up on this stuff. It's more beneficial to the security community as a whole to expose these things so you know how to implement effective security policies. When your trying to convince management to implement something security related it's one thing to say "We think this is a threat" verses saying "hey watch this" and demonstrating the threat.

Wow this is so useful for um security

If you pick the 5 billionth one you've probably picked a gooden.

We've banned this account for abusing the site, including posting many unsubstantive comments after we asked you to stop. Please don't create accounts to break the site rules with.

•These lists are for LAWFUL, ETHICAL AND EDUCATIONAL PURPOSES ONLY.

Yeah, like that is going to stop people from doing nefarious things with this info. If you feel the need to post this screechy, all caps disclaimer, maybe rethink your project entirely?

Geez.

Reply on news.ycombinator.com