royce

It's a fair point that raw password count is available.

But that value is an absolute number, without any in-API context of the total size of the corpus. This makes expressing relative rarity only possible by hard-coding the total size of the corpus into a calculation.

Put another way: the 20,000th position has a frequency value of "7889". But what does that mean? Where is that in the distribution of password frequency? It's impossible to tell, without manually constructed context that will change over time as the total number of passwords in his corpus expands.

But more crucially, there is no way to tell relative rank ("is this password in the top 20k?") using the API that I can see. That would make using the top X much easier. But with the K-anonymity "feature", there's no way to do that that I can see.

The purpose of a blacklist is multifold - to reduce the efficiency of an offline attack, in which the hashes are stolen and can be attacked at high speeds without rate limiting - as well as an online attack.

While Troy doesn't directly reveal the breaches, it is pretty easy to deduce which public leaks were part of v1 - especially once they're cracked:

https://cynosureprime.blogspot.com/2017/08/320-million-hashe...

I anticipate similar results for v2.

Indeed. One of the authors of 800-63B is actively involved in the password-research community, and is already aware that the guidance places no restrictions on blacklist size.

As a corpus to download for password research, this is indeed useful. But for providing a blacklist -- his stated purpose - it is not.

The crucial tell: his API does not allow the implementor to specify a frequency threshold (by top X in the list, or by Y number of unique uses of the password or higher).

By both API and explicit language in the announcement, he is promulgating the idea that checking the entire blacklist is useful, and "the larger the blacklist, the better." This is exactly what I'm arguing against.

And that's exactly why using this entire corpus - or even more than the first few tens of thousands - as a blacklist would be an extremely user-hostile choice [1].

Password psychology is remarkably consistent across a given demographic, because most people start by modifying of one or more base tokens that are already stored in memory because of their personal significance.

So unless the implementation gives specific, real-time UX feedback to teach users how to pick a password that is very unlikely to be in this (and future, ever-growing) versions of this corpus, using a large blacklist is "gotcha infosec". It creates UX where the user cannot possibly come up with a "good" password using most of their previous strategies. It's the worst kind of "gotcha infosec".

1. https://news.ycombinator.com/item?id=16434266

The UX of a blacklist with a half billion entries would be so crippling that it would cause a user revolt.

Most people's password-selection strategies are similar enough to other people's (like kbenson's 4000+ hit) that they could spend hours trying to come up with a password that has never been leaked before.

I tried to encourage Troy to suggest to implementors that blacklisting all passwords was a Bad Idea. Instead, he doubled down:

https://twitter.com/TychoTithonus/status/966400790221930496

Please don't use the entire list for blacklisting unless you actively also guide the user in how to generate a random passphrase (if a human must remember it) or a random password (if it will be stored in a password manager).

Instead:

1. Use a high-level password-strength assessment widget like zxcvbn:

https://github.com/dropbox/zxcvbn

2. Configure a blacklist with, say, 10K or 20K of the most common passwords.

3. Hash passwords with bcrypt cost 12 (adjusted to your platform's hashrate capabilities), scrypt, or the appropriate method from the Argon2 family.

But for all that is holy, please don't use Troy's entire corpus - or even the first million - as a blacklist. To quote the old NANOG saw, I encourage all of my competitors to use it. ;)

Edit: And since Troy's API at this writing does not support only querying the top X passwords, there's no way to use the API while avoiding the UX nightmare. So if you want to use this data in a professional manner, but don't want to download the entire corpus, here are the first 20K from his list (of which I've only personally cracked 19965 so far, interestingly; gist will be updated once I get all 20K):

https://gist.github.com/roycewilliams/281ce539915a947a23db17...

Edit 2: Preliminary results indicate that this data may be dirty. The 273rd most common password, according to Troy, is '$HEX'. This is almost certainly an import/conversion artifact, since the '$HEX' prefix is how most cracking suites escape non-ASCII or passwords that contain colons. I expect that there will be more artifacts. Use the data with caution.

For most people, a house payment also represents a direct impact to their ongoing cash flow. If your house payment is $2000 a month, paying off your house automatically increases your cash flow by that amount.

What else could you do right now with that cash flow - not only in direct alternative financial investments, but in indirect investments in your own well-being and quality of life?

Another option, once you have some cash built up and are otherwise diversified, is to occasionally request that the loan be recast.

https://www.bankrate.com/finance/mortgages/what-is-mortgage-...

Recasting works sort of like refinancing .. but your rate doesn't change and neither does your mortgage term.

Instead, a lump sum that you provide is deducted directly from your principal. Your monthly payment is then recalculated based on that reduced principal amount.

The financial institutions usually impose limits (minimum lump-sum amount, how many times per year you can do it, etc.) And they don't advertise it or make it easy, because there's no money in it for them.

And since the principal/interest ratio is front-loaded (so that much of the money that they make from you is in the first seven years or so of a 30-year loan), then if you can recast early in the life of the loan, the benefit is increased. (And that's the real reason why the industry advises upgrading every seven years - that 30-year, front-loaded clock gets reset in their favor.)

Even that is insufficiently precise.

Death is not the only thing that separates children from parents. Neglect, disinterest, self-involvement, inability to provide, etc.

And beyond a certain minimum viability, no adult is strictly required. Street "urchins" take care of themselves and each other.

eBay sometimes sends out emails with auction countdowns in them that do something similar.

The first time that I saw an accurate countdown ticking away in an email, it surprised the heck out of me.

You're welcome! And "¿por que no los dos?" :) I'll keep the gists updated and will also do a blog post, I think.

With a tool like hashcat, a modern GPU or two, and some publicly available wordlists, you can get the vast majority of them without breaking a sweat.

In other words: there is almost no value in hashing them with SHA1.

I've cracked* just under 99% of them so far (including the 14 million added in Update 1). Statistics are here:

https://gist.github.com/roycewilliams/b1de2afbfe5cb71bea16c9...

Regardless of composition, the top 12 lengths are:

    8: 32% (102260862)
   10: 14% (45084047)
    9: 13% (41525797)
    7: 10% (33632055)
    6: 06% (20211176)
   11: 05% (18275968)
   12: 04% (14052958)
   15: 02% (8291459)
   13: 02% (8042452)
   14: 01% (6321198)
   16: 01% (4201765)
    5: 00% (3054291)

In other words, requiring a minimum length of 12 would make 80% of the passwords in the corpus inapplicable.

... and the top 12 masks are:

  ?l?l?l?l?l?l?l?l,47823614
  ?l?l?l?l?l?l?d?d,7005728
  ?d?d?d?d?d?d?d?d,6212778
  ?l?l?l?l?l?l?l?l?l?l,6023602
  ?l?l?l?l?l?l?l?l?l,5379482
  ?l?l?l?l?l?l?l?d?d,5169013
  ?l?l?l?l?l?l?l?l?d?d,5090400
  ?l?l?l?l?l?l?l,4998896
  ?d?d?d?d?d?d?d,4798329
  ?l?l?l?l?d?d?d?d,4798124
  ?d?d?d?d?d?d?d?d?d?d,4754401
  ?l?l?l?l?l?l?d?d?d?d,4377841

Almost 48 million of them are 8 lower-case characters.

* And to be clear, "cracked" is an overstatement. Many of his sources are public. Simply using those sources as wordlists makes "cracking" these like shooting fish in a barrel.

A fantastic overview - clear and informed. Thanks very much for this.

Threads discussing rainbow tables are not applicable.

These hashes are not unsalted MD5. They are md5crypt ($1$[salt]$[hash]), as found in many Unix-likes and some Cisco IOS.

Twitter OP here. I know the thread's been flagged, but for anyone who finds the thread: he's OK.

https://dennisforbes.ca/index.php/2017/02/02/update/

Inside every set of debugging print statements is a debugging verbosity level trying to get out.

I argue that hash substring search activities (vanity hashes, partial collisions, full collisions, etc.) should be brought into the major password-cracking projects - economy of scale, etc.

In a way, password cracking is a specific subset of hash substring search -- in which we are searching for the entire hash instead of a substring.

This might be useful for that: https://scans.io/study/sonar.ssl

I personally attended a talk presented by Brian Behlendorf at the ISPF (an early ISP conference) in 1999 (which predates the 2000 Linux Magazine article). In his talk, he used the "A Patchy Server" explanation. He made no mention of any other etymology that I recall.

So either Brian was deliberately propagating the myth himself ... or maybe there's some retconning going on.

I'm not a math person either, but here's some fodder for someone who is.

Mark Burnett's extensive password collection (which he acknowledges is skewed, because it's largely based on cracked passwords, he only harvests passwords between 3 and 30 chars, etc.). Here's how some of his stats shake out:

* Although my list contains about 6 million username/password combos, the list only contains about 1,300,000 unique passwords.

* Of those, approximately 300,000 of those passwords are used by more than one person; about 1,000,000 only appear once (and a good portion of those are obviously generated by a computer).

* The list of the top 20 passwords rarely changes and 1 out of every 50 people uses one of these passwords.

So it's conceivable that 6M unique passwords could cover a very significant portion of a 120M user namespace.

Ref: http://xato.net/passwords/how-i-collect-passwords

A stock JtR 1.7.9-jumbo5, using the default rules, is finding quite a few of the non-zeroed ones pretty quickly. This surprises me; I would have expected them to have run the list through the JtR mill before passing it on to others.

You might also want to take a look at LOCKSS (I'm not affiliated; just thought it was interesting):

"LOCKSS (Lots of Copies Keep Stuff Safe) is an international non-profit community initiative that provides tools and support so libraries can easily and cost-effectively preserve today’s web-published materials for tomorrow’s readers."

Currently OpenBSD-based, but I think that I heard that they are looking to migrate to FreeBSD to take advantage of ZFS.

http://www.lockss.org/lockss/