CircleCI security incident | Hacker News Reader

CircleCI security incident

status.circleci.com

108 pointsmarkhelo45 comments

Is there any evidence there has been a breach of CircleCI directly, or just that MongoHQ suspended all access to their database(s) and so the service went down?

At this point it doesn't seem like they were 'hacked'[0], but simply involved in a larger security incident.

[EDIT] seems the linked page doesn't mention 'hacked' anywhere (literally - I searched for it!) Seems like the title[0] is a classic case of editorialising

[EDIT 2] As per below, would be very interested to see if their database being 'compromised' means that data was actually accessed by the attackers, or if they are just fall out from the bigger attack

[0] original submitted title was "CircleCI has been hacked"

>We are contacting you to inform you of an ongoing security incident affecting CircleCI customers, as a result of the compromise of our database (http://security.mongohq.com/notice).

Seems like a 'security incident' is code for 'hacked'.

I think I agree with you, however there is a big difference between having your service stopped as a precaution and having data lost/accessed by a 3rd party.

From that line it does seem more likely that they might be one of MongoHQ's clients who MongoHQ noticed had had their data accessed as part of the bigger breach.

In that context, saying they have been hacked is probably fair. Would love more details.

Up next: CircleCI customer announces data breach due to tokens accessed via the CircleCI breach (via the MongoHQ breach).

This is very likely. I'd guess that the likely target was a MongoHQ customer, and that MongoHQ getting hacked was collateral damage as part of an attempt on one of their users.

This is why I'm so mindful about using third-party services that require access to our code to perform their service. It's not that I don't trust the third-parties not to abuse their power, but by trusting them with my code, I also have to trust their security measures.

As such, I always very carefully weigh the benefits such a service can provide against how much work it would be to do it myself (which will likely produce a shittier result, but which will be fully under my control).

It's not even that I think I can do security better than $SERVICE, but I'm a much smaller target. I'm not as afraid of a targeted attack than I'm afraid of somebody compromising one of these focal points and then just lifting everything.

Quick theoretical senario...

User signs up to try circleci for a private project of theirs. Grants read access to their private repos via github oauth

User also has many other private repos (company they work for, open source projects, forks, etc)

Could they have used the stored github credentials from circleci to clone every private repo in full the user had access to?

In theory it is possible.

Github has a feature to allow access to a singular repository via a key. It would be logical for CircleCI to use that feature, although I'm not sure they actually did.

https://help.github.com/articles/managing-deploy-keys#deploy...

They did not.

They did, Circle-CI client here.

Info have a Circle-CI deploy key per private repository (which I will revoke).

I'm pretty sure that Circle uses the oAuth api to checkout repos, the deploy key part on github they use for their deployment feature.

If the attacker has a bunch of tokens, could they have bulk downloaded source code before the oAuth stuff was revoked by Circle?

https://github.com/blog/1270-easier-builds-and-deployments-u...

I am contemplating whether MongoHQ should just post their customer list, so their customers' customers know what to expect?

I know that's not exactly proper but... at this point, the hackers are way ahead of the users of sites that happen to depend on MongoHQ.

That could be irresponsible as a company which uses MongoDB for non-sensitive data (say data in publicly available pages) would be shamed when they didn't have a data breach of sensitive data. Or worse, a company that has signed-up but only played around with MongoHQ without using it in production. Each company should announce its own breach (and the scope of the data exploited) to its customers.

I understand all the implications you mention, and that's why I phrased it the way I did.

Though, the assumption I'm making right now is that the hackers know exactly what sites they've compromised. The longer they have this information in advance of MongoHQ's customers' customers, they more damage they can do. That sucks terribly for those people (potentially)

All this said, a serious lesson about PaaS/SaaS/DBaaS--whatever you want to call it--has been learned today.

[edit] - I'm reminded here of the Epsilon breach. Epsilon is magnitudes larger than MongoHQ and they apparently did decide to announce the names of affected companies. I am not sure how they handled that internally. [1]

[1] - http://mashable.com/2011/04/03/epsilon-hacked/

And an even bigger lesson to make sure you have strong permissions which can be quickly revoked.

In Epsilon's case those end users had direct consequences - their names and emails were compromised.

The end users of sites/apps/services using MongoHQ are probably unaffected in many cases since "a database" doesn't imply credit cards or even credentials or emails. The average database is probably just holding content for a site or app.

In probably close to all cases the end users would have no clue how they may be affected since they won't know what's in the databases, or what a database is.

This is one of the reasons why we have shutdown our own CI solution. We didn't have enough manpower to develop the product, audit the whole stack and monitor against attacks.

CircleCI's response to the incident appear adequate. In the best case, their system was shutdown before any keys were compromised. If so, that level of monitoring is certainly better than what the average business would be able to roll on their own.

from the page:

Identified - We have notified all users and recommended appropriate action:

"We are contacting you to inform you of an ongoing security incident affecting CircleCI customers, as a result of the compromise of our database (http://security.mongohq.com/notice).

We are taking aggressive action to protect your data and systems. At this time, we have suspended all CircleCI account access, and all builds & workers have been suspended. In addition we have revoked all access to Heroku and GitHub OAuth tokens and API keys uploaded to CircleCI.

We do not yet know the scope and impact of the intrusion and are therefore treating this event as if all data has been compromised. While we have no evidence that these credentials have been compromised, we urge you to revoke the following:

SSH keys that were uploaded to CircleCI API tokens added to CircleCI as environment variables secrets stored in GitHub repositories We will be keeping you informed at http://status.circleci.com and will update you at regular intervals as the situation progresses.

We deeply regret that this has happened and are working around the clock to resolve this incident and protect your data and systems." 22:25 PDT Update - We are still investigating the issue. The full team is engaged and we are working with upstream providers to diagnose and respond to the issue, and protect all of our users. We will keep you informed. 21:17 PDT Update - We are currently investigating an ongoing issue with our database service. At this time, we have suspended all account access to our service. All builds & workers have been suspended. We will have another update in the next 30 minutes. 20:20 PDT Investigating - "CircleCI is experiencing technical problems. We're investigating and should have an update within 30 minutes." 19:30 PDT

> variables secrets stored in GitHub repositories

Does this mean our code is compromised too?

That would seem to be the implication, assuming the people who hacked mongohq started trawling through circleCI's data.

If they have your keys then they can get at your code.

Talk about domino effect.

Our stuff, well non test/develop, is gpg'd in the repo. So should be fine.

Guess: this + mongohq was a targeted attack, aimed at a single customer of theirs?

I do not think so. There is currently no evidence that CircleCI itself had its data directly compromised as part of the MongoHQ hack.

They are just being cautious and assuming the worst and recommending their clients do the same (quite right too).

However the fact that so many apps can be screwed by a breach in a "Database as a service" style setup will make me wonder "How is this SaaS storing my data? internal or oursourced?" When evaulating new ones.

[EDIT] - Just noticed that I did not actually give opinion on the guess. It seems a reasonable guess that this could have been part of an attack on a MongoHQ customer.

I don't think it's relevant that it's a database as a service over all the other hosted services we put our trust in - if someone hacked Heroku, RackSpace or whatever other service provider they would get some db access, aws keys, source code etc too.

Private repos on github and bitbucket are probably a goldmine of accidents waiting to happen considering how many api keys etc slip into public ones!

Not really sure what the takeaway should be for developers building on [anything] other than to do your best not to store stuff in a way that can hurt your users if a platform you use is compromised.

I'm curious why they mention uploaded SSH keys. I presume they mean cases where SSH keypairs have been uploaded? The public key is public, and in fact anyone can use the github API to pull the verified public keys for any user in the system, http://developer.github.com/v3/users/keys/. If there is code uploaded with private keys in it, then it's likely there are other security problems in the organization.

Because CircleCI stores private keys for the sake of deploying to your servers when your build passes.

It's separate from your actual codebase having any private keys, which I agree would be a Really Bad Thing.

I think it's more like MongoHQ is compromised and consequently, circleci is affected than 'circle has been hacked'.

http://security.mongohq.com/notice So this really is mongohq's fault... Demonstrates the dangerous domino effect if you think of all the code/keys/secrets...

nightmares will be had.

Since circleCI users upload the private heroku SSH keys to automate deploys, how hard would it be for the intruders to get access to the environment variables stored in heroku?

Cache to their homepage: http://webcache.googleusercontent.com/search?q=cache:1oo0VIR...

Customers: stripe, Red Bull, Kickstarter.. and MongoHQ. How meta. This hack could get a lot more interesting depending on how much code they could have obtained from those companies.

I started testing out CircleCI a bit ago and this really doesn't help seal the deal for me. What am I supposed to do now? CircleCI was connected to my github account...are my SSH keys not safe now? If so, not clear on what I'm supposed to do.

Revoke the CircleCI key that you added to your repo on Github. In the worst case scenario, that key has been compromised and could be used to access your source code.

I didn't create a new SSH key...I just used my public rsa one. I should just delete that key and create a new one?

Is there any general sense of the techniques being used in these cracks? Just from a defensive coding perspective, I'd like to know if there are programming errors, weak links in frameworks, or what.

If you read MongoHQ's extremely detailed report[0], you'll see that the original intrusion was based on shared credentials (that were cracked on another system). This was then exploited in a number of ways to compromise various clients of MongoHQ.

Lots more discussion over at [1].

The main takeaway is to take security seriously, and employ multiple levels of security. The MongoHQ team are doing things like 2 factor auth, and restricting customer service tools to a vpn. As far as I can see, no framework or coding bugs.

[0] http://security.mongohq.com/notice

[1] https://news.ycombinator.com/item?id=6637426

Oh boy, another MongoHQ casualty. I wonder where the fallout will end...

You think this was a casualty, or the whole reason?

Well they also used Buffer's data for spamming, so it's hard to say whether one or the other was targeted or opportunistically attacked.

CircleCI is certainly the more interesting target as they have access to numerous companies' source code.

>access to...source code

How much do you think that is worth on the black market?

Honestly, not much.

You might be able to find a few vulnerabilities... maybe some good ones, maybe some hard coded keys, but about that's all.

Having a companies code doesn't mean you can make a successful business out of it. Especially if they are already established as a major player.

Was Adobe one of their customers? :P

any suspicions on who might be behind these attacks?

was the buffer spamming incident just a way to deflect attention of something more sophisticated?

I wonder if anyone used the keys stored in circleci to get to the code of some service and if we'll see even bigger hacking news in the near future

</conspiracy>

if you're using stripe, make sure to recycle those keys if you have them checked into your repo

Better yet, if they are checked in, rotate them and don't check in the new keys. Make them an environment variable instead.

Reply on news.ycombinator.com