1.4B records from “Have I been pwned” for analysis | Hacker News Reader

1.4B records from “Have I been pwned” for analysis

troyhunt.com

165 pointsrunesoerensen37 comments

Hmm. Given the amount of information removed (justifiably so), there isn't much left to analyze other than aggregates. (Which the dashboard mockup in the post already does)

What might be interesting is graphing a network of the relationship between services which suffered a breach. Fortunately, I have the proper workflow prepared to process the data in this manner, so I'll take a look.

You know, this is exactly where my mind went, too: http://i.imgur.com/fME7MK7.png

The difference between your first draft[1] and mine is that I've only included links where each site had at least 1% of users in the other site. This helps make links more meaningful.

For instance, about 37k users were members of both Tumblr and Plex. Tumblr had the largest number of Plex users, so it should be a strong link, right? No: because Tumblr has such a large userbase, this is a relatively insignificant relationship; 0.14% of Tumblr users had Plex accounts, which ranks only 20th for Plex.

Compare this to Plex and Xbox-Scene. 5.6k users had Plex and Xbox-Scene accounts, which is about 2% of each site's userbase. I believe this link is much more meaningful than the Plex-Tumblr link, and I believe my visualisation bears this out.

[1]http://i.imgur.com/FpLiFGk.png

* Note: since I ignored all lines with only one site, this "1%" only includes users who were in more than one breach!

Yes, that sounds like a sane optimization compared to my black-gray mess.

Yeah. I would've rather had a dump of just the passwords -- no usernames, site names, service names, counts, deduplication, just 1.4 billion passwords, one password to a line.

I'd feed it into my bad-password-indexing program (https://github.com/robsheldon/bad-passwords-index) and it would spit out a file that could be used to improve password security in a not-entirely-stupid way somewhere.

Also that's exactly what malicious actors would want to improve their dictionaries, to brute-force poorly-hashed dump from the next breach.

That's a tool they already have plenty of access to. I've tried to make a tool that will work against them, making it possible for any service, running any software stack, to efficiently blacklist hundreds of thousands of common passwords.

I can (and should) start tracking down more password dumps and compiling them myself to do the same thing, but I've been spread pretty thin for a while. It's been on my mind but there hasn't been time for it.

A massive list of plaintext passwords from Troy's dump, the ones that were available anyway, would have been really convenient. Odds are my index file would already be updated to a newer version.

I thought all his data came from publicly available dumps. If so, it might make it easier to for them to have it all in one place, but it's not like he's giving them anything they can't get if they really wanted it (and probably have already).

No, it also contains non-public dumps that someone gave to him (after authentication) iirc. But he never pays for dumps. I think he posted an explanation on his blog a while ago.

There are enough plain-text password lists for that, rockyou (old but good) and there are many more. Kali distro has some of them shipped if you are interested, and many more can be found online.

It's not all plain-text pwd's ofcource; a lot of the records (thankfully) have had hashed passwords.

Alright, made a first draft illustration of the relationship: http://i.imgur.com/FpLiFGk.png

I might be onto something. It appears the communities algorithm correctly detected the Russian and China communities.

So it would appear the most interesting data point on these records is the date and time of the breach discovery, or using that to estimate a breach date? Correlating that with some other publicly available data sets (e.g.: historical routing data) might yield some useful results?

I used to be somewhat interested by stats on passwords, etc. from breach data dumps.

haveibeenpwned is a helpful and legit site, though I think it should have used email confirmation instead of requiring only an email address.

I also respect Troy along with many other security researchers. Even those that are up to no good in the security world in some ways have contributed good things; after all, the rest of us are stronger and more vigilant now than we used to be because of their work.

However, this anonymized data will almost certainly be used by black hats more than white hats, and I don't see how this release is good for the majority of those that were affected by these breaches.

It does require email confirmation before showing "sensitive" data, like if an email was in the Ashley Madison or AdultFriendFinder breaches.

Which in and of itself is silly. The raw dumps are already available to everyone, blackhats included. Personally I'm tempted to make a site that just lists it by domain and stuff. I found several people at my company with Ashley Madison accounts using a quick grep.

Even so, raising the barrier of entry to this data will prevent some people from casually looking up their peers. It's worth doing.

Casually looking up your peers is exactly what you should be doing in my opinion. But I'm not a good person and I'd rather see details like that plastered everywhere. Be an idiot, get what you deserve.

I have a 2004 Gmail account which people like to use as a fake email address when signing up to things. So my address is in the Ashley Madison leak. This is not something I'm comfortable with people knowing without context, so even if it was a policy of "play stupid games, win stupid prizes" I'd get harmed.

You must be a blast to work with.

I'm with you that haveibeenpwned should require email confirmation before listing the breaches an email address was in. As it is, it has become an easy to use dictionary of places to find personal data for any given email address. Yes it doesn't list the breaches that Troy has deemed "sensitive", but if you're trying to commit identity theft or fraud using personal data, it doesn't really matter if these search results don't include websites that deal with porn/adultery/children.

Doubtless there are black market tools that provide such a service but expose far more data, but haveibeenpwned lowers the barrier to entry significantly by being far more available to the public.

How will black hats use the data?

I have wanted to ask this for a long time. What do / should you do once you have been pwned?

You definitely have to change password, or even use password manager. But your record is now widely available it feels as you have been naked on the Internet.

You may likely change your email address ( Login Name ). But opening another email account is such an hassle. An opening yet another account on those of your favorite site means you lost all of your pass record.

Saw this on HN earlier in the year and have been slowly phasing into my life. I have mail setup for my domain and assign per-site aliases (hn@mail.com, fb@mail.com, etc). All of the mail just forwards to my gmail that I've had forever, though that is more of a matter of convenience and habit. The key thing is, I can spin up a new email whenever I want since I am the wildcard, and I control the host and am not tied to the whims of gmail so I am covered on both ends.

Be careful running your own email server. Some domain hosts have terrible security (easily socially engineered).

I would love to know which DNS provider has the best security if anyone has done the research.

Running your own name server is even easier than running your own mail server.

Running your own registrar gets tricky, however.

But registrar still have access to change to what NS domain points. (and DS records)

What are you worried about here? Further breaches? If you've changed your password, there's not much more you can do. Getting a new email address doesn't make you more secure. If you're trying to conceal the fact that you have an account somewhere, you should not be using your email address there in the first place.

I'm always amazed at how efficient AWS is at detecting private keys on the internet (checked into github, etc..), and then proactively locking down accounts. I wonder how long it will be until we see a similar service from consumer accounts, like github or twitter. Seems like "have I been pwned" might offer a commercial API for such benefit...

If the bots can find the api keys then amazon can just run their own bots.

HIBP is simply a database of email addresses that have been associated with leaks. I don't see how that would help him identify private keys.

Well AFAIK Twitter doesn't give users keypairs, so I think they meant do the same with leaked credentials.

yeah, that's what I mean. Provide a service that reports back leaked working credentials so that the service provider can lock down the account.

The data can all be found here as well though: https://www.thecthulhu.com/

I've come to assume some non-zero percentage of sales/marketing/lead gen companies are using this as a go-between for an SMTP validator.

Awesome Analysis of the data - https://www.sizzleanalytics.com/Boards/sizzle/Data-Breaches-...

I still want the dumps for myself, so I can determine if any of my accounts are compromised. I understand why it's not possible to distribute these dumps, but for the same reason, I'm not going to search someone else's database. How can we have a database of dumps that people can query safely, without revealing their information to the people with the dumps?

Search the dump yourself and look for your address.

Or even better, build the service you're asking for! Clearly, there's a need.

Good distinction, data for analysis/trends, data for nefarious victimization.

Reply on news.ycombinator.com