CircleCI trusts 8 analytics companies with your source code and API tokens
kev.inburke.com
kev.inburke.com
We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.
CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company.
If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988.
As a customer, Rob, what would be a good avenue to notify you of issues like this in the future? Not every company wants to be front and center of what is essentially the news hub of tech people like HN for issues, so is there some way we can get with you guys directly to straighten security issues out, as well as some assurances as how you'll handle it along the way?
Thanks for your attention to the issue :)
You can always email our security team at security@circleci.com Our gpp key is available on our site https://circleci.com/security/ if needed. You can also email me directly at rob@ if you have an immediate question.
You mention vetting third party scripts, but did not explain how your app is not vulnerable to those scripts being hacked or modified in the future. This vulnerability seemed to be the main point of Kevin's piece.
Could you update your post to address that issue?
p.s I'm curious why you thought this route was better?
It's a corporation... Why should he care about it's feelings? What does he owe CircleCI? Are they paying him? What claim do they have to his time to jump through their hoops / process?
I'm positive in this case that they weren't intentionally messing with people's security, but shouldn't we and their customers be able to judge that ourselves instead of getting it swept under the carpet via private channels?
I do believe it's good practice to "not be a douche", especially at a personal level, but I wouldn't even come close to categorizing this as douche behaviour.
* Note: this comment is not actually directed at CircleCI, just the attitude that we all somehow owe it to companies to tell them about their goofs privately.
A typical case for responsible disclosure is something like a bug in Apache or Nginx that's serious and if a security issue is found that they're given time to address it. So when they make an announcement its:
1. Here is the issue
and
2. Here is the fix
Instead of just spreading and publicizing a vulnerability.
There's been some really high profile breaches where the extent of the impact to the customer has taken an unacceptably long time to be made public... which results in increased fallout damage.
A company's natural inclination is going to be to minimize losses (generalizing of course), and that often means dragging their feet, or dealing with it privately and never notifying their customers.
We should recognize these incentives and be a bit more aggressive about holding their feet to the fire, instead of criticizing the researcher who discovers the security flaw. Unless of course his / her behaviour is blatantly malicious.
It anecdotally feels a bit lopsided in favour of the corporations at the moment. Especially when someone like in this post, who clearly didn't endanger any customers, gets criticized...
Also, it's not swept under the carpet - it ends up usually getting $ for the reporter and a better story as they'd also know how the company fixed it. If they refuse to fix, then publish away.
I think we can rule out a malicious decision regardless though. I'd wager if it was flagged by the annoyingly pedantic but super smart developer, then it's still sitting in their Trello board buried under a few hundred features that were considered a higher priority by the product manager. In this case, the decision felt mostly harmless.
Or yeah, probably just as likely that no one noticed.
Edit: Okay, I get it. People like convenience over security. I'll stick to self-hosting a Drone-CI instance and only deploying binaries.
Most of them.
This isn't the same as a browser vulnerability where some kid could hack a load of people before they patched. One of these supposedly trusted companies could attack your customers... But they could already. They know their scripts get loaded into all sorts of inappropriate environments.
Getting people notified, getting credentials changed. That's the paramount concern. You can rub PR lotion into an in-depth audit later on.
It's an industry wide bad practice and a risk, but forcing password changes or notifying people is quite frankly ridiculous.
Many of their clients may treat this as they would an actual breach. It's somebody they haven't vetted having potentially complete access to their development chain and production secrets. They won't know until they look. They won't look until they're told.
And what's CircleCI paying a for this breach got to do with the price of fish?! Say you hired me and gave me full access to everything in your business. Then one day I turn around and tell you my extended family, my friends, my dog walker and my cleaner have all also had access to that data. No big problem eh?
I count it as "inappropriate and unauthorised access to data". Where "access" is potential, not necessarily actual unless you can absolutely prove there was no access.
These third parties have had access to sensitive data they shouldn't. That's a breach in my book.
Neither you or Circle CI even can say this hasn't lead to current or past third parties —or their rogue developers, or people who have hacked them— gaining source access or customer data from Circle CI users. Why? You simply don't know what was running at any given point.
Auditing and sub-resource integrity would help in the future, but it's too late. Unknown people have had access. Only the Circle CI users will know what ramifications that could have on them.
If your argument is anything more than a redefinition of "breach", please explain why you're being so nonchalant (and why you think I'm being "ridiculous") about this.
The client vetted CircleCI, and CircleCI presumably vetted the third parties. It is not fair to say these vendors have not been vetted.
It may not be a best practice, but it's little different than CircleCI (or any other company) contracting with a private data center, which has direct physical access to their equipment. They have presumably vetted the data center provider, or cloud computing vendor.
It would be nice if Github offered more granular permissions.
https://developer.github.com/apps/building-integrations/sett...
1) Create a new GitHub user: e.g. (circleci-builder@example.com) 2) Grant read-only access to specific repositories to the new user. 3) Configure CI to use that user.
I know Gitlab is HN's darling but to me it's not reasonable to plug your service every time a competing service is mentioned.
You can not opportunistically spam nearly every thread where GitLab might be tangentially related.
I don't use GitLab largely because of the way you post on HN, though the software seems reasonably well-written. It's very frustrating.
> To be clear, letting third party JS run in a trusted environment like a dashboard is an industry wide problem. If we assume CircleCI is the only bad actor we're kind of missing the point of the exercise.
Ok....
It's informative, not "gitlab doesn't suffer from this, here's a 10% off coupon https://gitlab.com!!".
More generally, however, you're right although I have just come to accept that this is how things are on HN. Choose pretty much any "Show HN" thread or any thread about some cool new app/product/service and in the thread you'll find a few "shameless plug" comments that would be considered spam in any other forum: "Hey, we also make an app that does $foo, sign up for our beta at example.com and check us out!".
(One of the worst "offenders" is Userify. Pick any thread that relates tangentially to user authentication and chances are good you'll find a comment from them that manages to work their name/URL into the conversation somehow.)
I also think the CI service, which GitHub doesn't provide, is a higher-risk environment for this kind of thing. GitHub settings pages don't production deploy keys.
(I have no relationship with Github other than that I am a customer and have watched them steadily hire some of the best people I know in the industry).
Excuse me, is there any article about this, or maybe some pointer where one could get a GA script that doesn't need `script-src data:` (or eval or similar insanity) in the CSP?
I've tried to add CSP for a page that has GA (no other external deps) and it seemed to deliver some scripts from base64-encoded data URI. I haven't researched what exactly it does, but suppose it was the unpacker inserting code that way, instead of using eval. Could be wrong, though, but the only external JS reference was analytics.js, and when testing in Firefox 57 CSP had complained about script with a data URI.
Since you're asking though, here's one thing you can do better: Stop spamming HN with your irrelevant nonsense.
The person that I responded to said something about GitLab so I told them how I felt about their comment and their product. I would've said the same thing in person.
Nearly everyone on HN has heard of GitLab. They don't need the astroturfing or spamming you are accusing them of.
No because then the comment would be on-topic.
> GitLab is not some random company spamming their crap.
Yes they are.
> Nearly everyone on HN has heard of GitLab.
Because of the spamming.
A few big site operators should push back against Google on this. The New York Times and CBS, for example. They use Google Publisher Tags, but no Google ads, so they don't really need Google on their pages.
Q: Is it violating program policy if I place ads on iframe webpages in my software?
A: Yes, it does violate our policies. Firstly, you’re not allowed to place ads in a frame within another page. Exceptions to our policies are permitted only with authorization from Google for the valid use of iframes.[1]
None of that has to do with how ads from their network are actually rendered when you use their tags. Adsense/DFP tags do use iframes for almost everything unless explicitly set to load directly on page, usually for legacy code or rich-media. The relevant help article is here: https://support.google.com/dfp_premium/answer/183282
"...SafeFrame is supported in DFP and enabled by default when using Google Publisher Tags. Reservations serve into SafeFrames by default, but you can disable this setting...and use friendly iframes instead, if needed.... Some creatives, such as expandable ads or creatives that access your page’s DOM elements, might not render correctly in SafeFrames or other cross-domain iframes. We recommend updating these creatives to make them SafeFrame-compatible, in order to retain SafeFrame’s security benefits. If this is absolutely not an option, there are a few things you can do to allow reservation ads of this type to render properly..."
Either way, iframes are common and the ad industry is not the issue here. Many analytics providers that track all events automatically do need window access and the normal install for all of them is tag on site.
I wish more apps supported this.
Also on my list is a "Don't ever let me disable 2FA" setting. I'm more worried about malicoius resets than I am about ever losing my 2FA device and backup codes.
> Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these.
I surprised that secrets make it into the logs at all. I would have expected them to filter anything that's in an env secret from the log output. Pretty sure Travis does this.
> Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain.
That's not much of an option for this type of thing as the third parties would then have a PITA time dealing with upgrades (so they wouldn't support it). Using <script src="hxxps://tracker.example.com/path/to/script.js"></script> (rather than a fixed version) allows for live upgrades as they're in control of the resource that is loaded.
And you can't load the script from your domain as at the end of the day it's got to get instructions and upload data to the third party. At best you'd be loading a shim that does the same thing as the script tag.
I like this idea. I'd only want to use this though if my Google 2FA backup codes are backed up via Authy or a similar app.
I don't trust codes I've printed out and have always lost.
Wouldn't this be possible to circumvent if the user is allowed to switch TOTP devices? Are you saying you'd like a way to irrevocably tie this to a single TOTP secret? And you're not worried that someone could steal your TOTP secret and you'd be 100% powerless to stop them?
Changing devices is allowed but to do so I would authenticate using my password + 2FA prior to doing so.
Video games have been doing similar things for character deletion for years, yet it is rarely if ever found for account credentials in general.
It does.
Pusher is a SaaS product for websockets and push notifications. I think it's reasonable that a company like CircleCI might outsource this instead of maintaining the infrastructure for it in-house.
Launch Darkly is a feature flagging service. While I feel the value is much less than Pusher (and I'd be inclined to just build an in-house solution), I think it's still a relatively reasonable tool to use to create a better user experience.
Intercom when used for customer communication is also something that I feel I'd want as a user, although this could perhaps only be loaded when requested by the user.
There should be a corollary to NIH Syndrome[1] that takes into account 3rd party APIs. The way I see it, it's not about reinventing the wheel, it's about having your house in order.
Whether a developer (or business person) made a business decision to outsource CI has nothing to do with technical competence.
Using SaaS for any business data has a risk, but usually it's worth it. I'm sure you use slack, slack could be breached, and I'm sure no one at your company has ever slacked a password to someone else.
Developers tend to trivialize this: we tend to only consider the initial setup time, and pretend no further work is necessary.
you can say that they are 'doing it wrong', but without execption the shops I've been in that use CI tools
o have one guy, maybe no longer with the group, that set up the CI deployment and no one else knows how to deal with it
o dont have a firm control of their development build and dependencies and the different CI environment is a constant source of shear
o are running a service with its own deployment chain, which differs in environment from the CI server and should arguable be used for tests
o dont have a decent way of running the tests outside the CI environment at all, which makes debugging CI failures pretty problematic
o because of the single maintainer issue, new tests often dont get integrated into the CI, which is really conterproductive
o for distributed services and services with runtime dependencies, the CI isn't really providing the whole picture (meaning we should really be using the delopment tools to spin up transient test instances anyways)
o the state in the CI often tool doesn't get packaged up with the repo, so it doesn't transition easily to other development teams
I'm very sympathetic to avoiding wasting time on home grown solutions (even if they are as trivial as 'run these 50 tests and collect status). but if the development effort is sufficiently small, or sufficiently complex, I think external CI is actually costing more than its worth, or giving people a false sense of comfort about their testing and dependency management. thats all without considering any trust issues.
That said, the same could be said of any piece in the stack: Why use a framework? Why use Github? Why use hosted email? Why use cloud? Why not hand new hires a blank laptop and a USB key with the latest Ubuntu?
That's what I got at my last two jobs (minus Ubuntu), and I loved it.
The important point is that there are now 8 attack surfaces instead of 1. Whether they're analytics or pad-string doesn't really matter.
That's the best case scenario. What if these 8 companies also give the source code to other companies?
Also, not that they do so, but would Access-Control-Allow-Origin set to something other than * prevent 3rd party requests to the API for scripts loaded from 3rd party domains.
Also curious if anyone has written a JS library that patches XMLHttpRequest.prototype to audit exfiltration of data in the DOM.
I have hit numerous bugs with their website as well where stuff doesn't load, builds kick off into infinity, when I transferred a repo everything broke, ack!
Furthermore, I am frustrated with their pricing, they go from free to $50/mo for the next upgrade tier, that seems like a crazy jump to me. I would gladly pay $10/mo or so for another container or 2x parallelization. The issue is being a single developer I rarely would save any time and the $50/mo is just too steep.
Finally, when I tried to build a justification case for my manager to pay the $50/mo, I wanted to use data from their CircleCI Insights which shows how long your builds are queued on average and some other important data points. But you cannot access these insights from a free account. Seems like that info should be available and prominent to help people understand the cost-savings they might receive by upgrading. I emailed support and asked for a one-time data point for that statistic to build the case as I was considering upgrading and they said sorry, nothing we can do for you. Is their goal to make money and have happy customers? If so they aren't doing a great job of it.
Overall a lot of frustrations with the platform and this just adds more fuel to the fire.
I do wish Jenkins to be a little less high-maintenance, like say having a postgres backend and a docker image for the master. I hate having a different backup routine for each thing.
To be fair, it was very stable, though UI was awful. Nowadays we use GitLab (selfhosted) and love it.
I have only been tasked with fixing setups on Jenkins, and a friend of mine has spoken at length about how difficult things are to set up.
Some of that is based on the difficulty in getting a Jenkins configuration into source control. For CircleCI, I know, the CI configuration goes in a configuration file that gets checked in to the repo.
Another big part is making sure the development environment is sane, and replicating it. When we were dealing with Jenkins, Docker didn't exist yet. I assume that build slaves running inside Docker are a thing now? Because that's another thing that CircleCI gives us.
Honestly, though, this is just a thing that's easier to outsource if you can. Running a Jenkins server on your own is valuable if you have business reasons ("Code cannot go onto a third party server!") or if you're doing testing that involves connected devices (build and deploy to this Android device, then run tests...). But apart from that or a similar motivation, why bother maintaining a server when services are available for fee or cheap?
https://github.com/drone/drone
We went from Drone -> CircleCI -> Jenkins in the last two years. Jenkins feel opaque and hard to reason about, periodically dies for no _good_ reason and the UI is atrocious.
Drone itself was fine, as a minimal feature set. I think the docs are offline now, but it's fairly straightforward.
You must be doing something wrong then. I run a Jenkins cluster with hundreds of automated pipeline build/deploy jobs and the master node never dies. (It's also the only node up and running 24/7) You should be offloading your work to build agents.
I was merely a poor steward in the early days out of necessity. My era of tinkering ended once we moved to CircleCI.
Upgrade to an additional container is just $25. \
There are some defenses that can be put in place. The first one is kind of awkward in many cases which is to host the JS on your own domain. There's still the risk of course that it will go off and get additional code from the 3rd party source to execute, but that can be reviewed for.
The other option is to use sub-resource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to ensure that only scripts you've reviewed are used.
Of course you need then to make sure you're notified before the 3rd party makes changes that would break the signature.
What kind of pushback do you get and how do you handle it?
To me, it's not really a debatable point that loading JS from a source you don't control implies trust in that source and therefore a risk that if they are compromised it affects your site.
Whether that risk is ok for a business depends on a number of factors like :-
- How trustworthy are the sources they're loading from? - What reviews have they completed on the security of those sources? - Do they have contracts in place with those sources that cover the requirement for security?
Doesn't API have seperate authentication credentials (API key/tokens) than the web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape what's rendered on the UI, but making calls to the API??? I wonder how that's possible in CircleCI case.
The only thing that 3rd party JS will have access to is everything on the page.
Now, the things on the page are sensitive (the secrets from the env variables are dumped in the UI).
Secrets in the secrets page are not exposed at any time, even when you go to edit. Only in the build screen, it's exposed to the output.
Those 3rd party libs DO NOT have access to your source code at any time. The only time they will have access to your code is if they gain SSH access to the machines that are running the build. And that's not trivial.
Even if your public key is exposed, they don't have access to checkout github with that.
So yeah, it's 3rd party libs in a "private" context but not more than that IMHO.
I guess we'll have to wait for a CircleCI statement to address this in detail (or someone testing it).
Therefor in theory other scripts loaded on the page could grab the token and make authenticated API calls as well? Although I guess this is mitigated by verifying script integrity when loading scripts from CDN (e.g. integrity attribute of script tag)?
I feel this is a bad way to approach it and a bad attitude in general. Showing people how this can be exploited (after responsible disclosure) is a great opportunity to spread awareness of potentially their own issues in the same space and create points of discussion. If a user feels compelled to do so, they can seek out the info. But to show disdain to the reader for their lack of knowledge comes off as unprofessional in spirit.
BTW this last easily misunderstood sentence sits just above your "I'm available for hire" link, which is not exactly painting you in a bright light for recruitment.
Today it's perfectly normal. I don't have exact numbers but at least 50% of all sites pull in third party code. It really is bizarre to me that this became the norm.
>Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how.
That leads me to this question: Did the author reach out to CircleCI to report this issue, before publishing their blog post?
What you as a commenter, or a company (usually prefers to quietly fix this), or the wider user base (usually prefers a 'heads up' immediately), all have different opinions about what they see as responsible.
Also, it doesn't seem fair to dismiss this disclosure as a 'hit piece' unless it is factually incorrect.
Write. I have written at least ten posts that made the front page of Hacker News / Programming Reddit (nb - Those aren't the best proxies for quality or popularity, but they are well known proxies). I can help kickstart your company's engineering blog, or work with your team on story/content ideas.
Now I'm not sure how much making the front page motivates your writing process. It is an interesting topic but I feel it would be a stronger case to demonstrate the issue across products from multiple vendors, though less incendiary/front-page-y.
PS. Props for not being the one to submit this particular article!
I used to be in ad tech, it always baffled me that we had first party js on every single customer page. Conceivable if we were breached, someone could inject code to read every login token or cc# or anything and send it to their servers.
Or is that a browser extension that the OP forgot about?
I’ve tried to explore the privacy aspects here [1] (like many, we completely separated libraries for the public site and libraries for the apps)
I think we need a framework when thinking about which companies to pick when adding 3rd party features to our products - and probably a system to trust such 3rd party with their data processing.
I’d be interested to know how companies make these decisions today...
[1] https://blog.getkumbu.com/posts/building-a-privacy-respectfu...
should get off their duff and analyze their own internal server logs.
If it's in the overall sales conversion funnel it will be optimized. Credit application flow was scrutinized just as much as adding to cart and checking out.
By installing uMatrix and not approving these things, you are (a) reducing the probability that a mistake, misaligned incentives, etc, would harm you, and (b) get a notification that there's something you wish to block.
uMatrix IS part of the real solution. The other part is avoiding providers whose security standards are not up to yours.
We need to consider the good of the many, and client side solutions like umatrix are not a solution for the many.