CircleCI's incident response | Hacker News Reader

CircleCI's incident response

blog.circleci.com

59 pointscr4zy11 comments

Ouch -- Amazing how there is this ripple effect happening now that everone is using the cloud and there are so many middle men or service APIs.

> In order to protect your data and users, we strongly urge you to secure each of these systems:

  > SSH keys uploaded to CircleCI
  > API tokens stored in CircleCI env vars
  > API/SSH key stored in a GitHub repo accessible from your CircleCI

Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems. So, someone could have used these keys to have much wider access!

> Interesting to note that Stripe and Kickstarter are customers. This is a little scary to think their source code could have been exposed, given the $$$ flowing through their systems.

If either of them have any sense (which I think they do!) then having there source code exposed shouldn't in and of itself pose a security risk. If it did then that would be security via obscurity.

Now I'm not saying that having the source code for a site is useless or that having it exposed is a good thing (it's not!). If there is a bug/hole you could find it, but if there isn't then the source being available should not make anything less secure.

> So, someone could have used these keys to have much wider access!

SSH keys and API tokens are another story. These need to be revoked immediately (CircleCI's response looks great by the way) as there would be nothing besides firewall rules (if they exist!) preventing an attacker from using them.

This whole incident is a friendly reminder to follow the principle of least privilege[1] with your XaaS providers. There's not much you can do for API tokens. If you're relying on someone else to provide a service on your behalf then they'll need their own token.

For data though it's possible to encrypt sensitive data on your end and use the DBaaS provider as an opaque blob store. Doing it across the board gets a bit unwieldly as it's hard to query encrypted data but it's fine sensitive details that you don't want to necessarily expose (such as ENV vars, payment info or equivalent tokens). If the provider doesn't need access to X, and X is sensitive, then encrypt it before you save it with them. That way you don't have to worry about them leaking it. One of the best examples of this approach is tarsnap[2] which considers everything sensitive and encrypts everything client side (on your machine) before uploading to its central server (and then S3).

EDIT: One more quick point. If you explicitly encrypt sensitive data as part of storing it (regardless of whether you're using an XaaS provider or running a DB yourself) there's one more big advantage: Backups!

Your backups themselves should be encrypted (ex: we use gpg prior to uploading them to S3) but in addition to that by following the above, the sensitive fields in the backup are also encrypted. This means that they're encrypted in any scratch/temp files that are part of the backup generation process.

Otherwise they would be stored in plaintext on your server and possibly be on your server's hard disk (or your XaaS providers disks) without you even knowing it. Oh and remember that disk blocks don't actually get overwritten when you "rm" a file so this is more important than you think.

[1]: http://en.wikipedia.org/wiki/Principle_of_least_privilege

[2]: http://www.tarsnap.com/

>source code exposed shouldn't in and of itself pose a security risk

It shouldn't, but it's easier to find and form exploits for services if you can inspect the code, versus having to poke around on the live site.

Stripe is in the business of fraud detection, just as any other payment provider. Their source code likely contains lots of automated heuristics checking for fraudulent transactions, information which is tremendously useful for criminals looking to defraud people. No need for the "security by obscurity" reflex, because the problem at hand is impossible to solve completely.

Now might be a good time for circleci customers to pick up a code analysis tool such as brakeman and look for obvious security holes in their apps. If the attackers have hundreds or thousands of web apps' source code, I'd expect them to start trawling for vulnerabilities with automated tools.

What happened between the 19th and the 28th? https://news.ycombinator.com/item?id=6638004

  Hi Justin. To clarify, from what I understand, October 28
  is the date MongoHQ detected this. They've provided us 
  with the logs of database access, and unfortunately the 
  queries leading to our spam attack on Saturday started as 
  early as October 19.

So, to be clear. I need to contact Github directly to see if my source has been downloaded using compromised deploy keys?

I would. Although you can see some of that info here: https://github.com/settings/security

Edit: There are also logs for your organization in https://github.com/organizations/<your_organization>/setting...

I emailed GitHub this morning around 8am EST. I received this response this afternoon. It sounds like they are doing their own audit and will contact you if they find anything unusual:

> I wanted to check in with you about this incident - access to repositories on GitHub is logged and we're currently investigating the potential of unauthorized repository access allowed by the MongoHQ and CircleCI breaches. Although we don’t currently have any evidence of unauthorized access, or specific news regarding your repositories.

> Last night, in coordination with CircleCI, we revoked all SSH keys and OAuth tokens added to GitHub accounts and repositories by their service. We'll be in contact again when we have a more substantial update or news about your organization's repositories.

If I had a heroku API key stored with CircleCI, should I assume all of my app's production config variables have been compromised?

Err on the side of caution and rotate your keys.

Reply on news.ycombinator.com