Important Security Notice from Patreon | Hacker News Reader

Important Security Notice from Patreon

patreon.com

59 pointsrdl20 comments

As always, these disclosures are an opportunity for all of us to learn something without having to pay the substantial reputational price for doing so. I'd encourage everyone on HN to read the How We Ended Up Here parts carefully and take simple remediation steps in your own projects/companies.

The unauthorized access was confirmed to have taken place ... via a debug version of our website that was visible to the public.

This is a ridiculously common way for attackers to get into your network. You both want to have an accurate inventory of the apps/boxes your organization has (highly non-trivial at scale) and you don't want dev/staging/etc to be publicly accessible. The simplest way is to firewall off traffic to them.

If you need them to be accessible for e.g. callbacks from public services, default-deny everything and control access at the network or frontend web server level before things get to your applications (which are likely going to be your vulnerable underbelly).

In most cases, the dev servers will not themselves have really interesting things on them. They will instead be used by attackers to pivot to other, more interesting servers on your network. Patreon's understanding is that the dev servers lacked SSH keys to the production servers -- that's a good thing, but after the bad guy has a dev server with local network access to interesting machines, they're going to do things like e.g. probe for exposed admin consoles to network appliances, outdated RedMine instances, internal tools, etc etc, and they will eventually own every machine that talks to any machine that your dev server can talk to. You will eventually lose the entire deployment if the bad guy wants you to.

The development server included a snapshot of our production database, which included encrypted data.

Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)

>>The development server included a snapshot of our production database, which included encrypted data.

>Never, ever do this. (It's been a firing offense at most shops I've worked at in my career, and that's for good reasons.)

Would you use a migration script that "scrambled" certain columns on the way back from Production to Development? That's the only way I've seen it done properly at $EMPLOYER.

Would you use a migration script that "scrambled" certain columns on the way back from Production to Development?

No, because you'll almost certainly fail to fully anonymize the data and if there is anything "interesting" in your possession it will probably still be recoverable. I wrote a comment on HN long ago about this, and still stand by it.

https://news.ycombinator.com/item?id=1142326

Wow, thank you. It took that kind of crystal clear example for me to understand the risk clearly.

Seeding development with production data is just lazy, unnecessary, and, as we see here, dangerous. Spend some resources on a proper seeding setup.

What would you suggest for such a setup? Also, I did not say seed it with production data - I mean jumble PII and then import a section of production data into dev. Sometimes this is necessary.

A while back a fake DB data generator was posted to HN: https://news.ycombinator.com/item?id=8761953

Further back, there's also this one for Python: https://news.ycombinator.com/item?id=7102268

And this one for PHP: https://github.com/fzaninotto/Faker (there were no HN comments on that thread)

And this one for Ruby: https://news.ycombinator.com/item?id=86018 (also no comments)

These seem like a good starting point.

Also, mining bug reports for data that causes problems in your application is a lot of fun. You can turn your development database into a collection of pathological edge cases.

I've worked for organizations where you could be disciplined for not stripping out certain data from a database before using it for testing - primarily SSN and email addresses, but it's absolutely necessary at times where using real data will allow you to identify the cause of an issue whereas troubleshooting blindly on test data could take 10 times as long.

For once we get what we need to know. They used bcrypt and everything is nicely salted.

Congratulations Patreon, you just executed a textbook "good" public disclosure of a security breach!

I agree. When i read this email this morning, I came away feeling more secure with the company, rather than less. I hope that more companies do it this well, and realize that admitting the breach well can strengthen the trust.

This is out and out ownage. Patreon has basically been Ashley Madison'd. Source code, internal dev documents, cron jobs, and the entire 13GB database is out.

My confidence is zero.

> unauthorized access to registered names, email addresses, posts, and some shipping addresses. Additionally, some billing addresses

Considering patreon has some users who have been, and still are, targets of harassment, this could be much worse than having their credit card or password leaked.

This guy: https://twitter.com/Tulpamania posted about the attack on 28th Sept...

I'm particularly curious how they protected SSNs.

> Although accessed, all passwords, social security numbers and tax form information remain safely encrypted with a 2048-bit RSA key.

> ...

> We protect our users’ passwords with a hashing scheme called ‘bcrypt’ and randomly salt each individual password. Bcrypt is non-reversible, so passwords cannot be “decrypted.” We do not store plaintext passwords anywhere.

Updated post now says the above.

Generally you don't encrypt actual data with rsa, you encrypt a symmetric key with rsa, then encrypt the actual data with the symmetric key with symmetric encryption. An obvious question is what granularity is the symmetric key for their data. A different one for each piece of data?

Usually rsa is used when the source of the data never needs to read the data again. Usually for example like sending an email. The sender encrypts it and sends it, but the sender cannot decrypt it. Why were they using rsa for the tax information? Were there multiple servers and the server that creates the information is not trusted with being able to ever read the information later?

AFAIK there is nothing stopping you from encrypting arbitrary amounts of data directly using asymmetric algorithms like RSA. The scheme you describe is implemented because asymmetric crypto is dead-slow compared to symmetric crypto, so the amount of data to be encrypted using an asymmetric algorithm is kept to a minimum by just encrypting a random symmetric key, which is then used to encrypt large amounts of data with a fast symmetric algorithm.

Since the data that was encrypted in this case is inherently tiny (social security numbers etc.), I'd think it would be a good idea to directly encrypt this with an asymmetric algorithm, thereby reducing the complexity of the entire mechanism, which is generally beneficial for security as well as maintainability.

But I'm wondering as well about the reasons for choosing an asymmetric algorithm in the first place. Typically, symmetric algorithms are used for such cases in which the entity that encrypts something is the same entity that needs to be able to decrypt it later. It's not that this could not be implemented as well using asymmetric crypto, but it somehow defies the whole point of the asymmetry.

Maybe what they meant is that they hash using bcrypt and then encrypt the hashes?

I just see a blank grey screen, because the page doesn't display if you have Facebook Connect blocked.

Why are they using asymmetric encryption for passwords, SSN, and tax information? I've never heard of that done.

Reply on news.ycombinator.com