Attack campaign involving stolen OAuth tokens issued to third-party integrators | Hacker News Reader

Attack campaign involving stolen OAuth tokens issued to third-party integrators

github.blog

281 pointsheyoni67 comments

If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.

I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it.

I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.

https://docs.github.com/en/developers/apps/getting-started-w...

All the oauth prompts I’ve gotten in the past 3 years or so seem to have the granular per-repo control you’re looking for.

Only "app installations" from the GitHub marketplace can be limited to specific repositories. For OAuth apps this is not possible sadly (everything that uses GitHub to login or connect accounts, e.g. Travis CI, Terraform Registry, Heroku).

I wrote about this a couple of years ago

https://games.greggman.com/game/github-permission-problem/

Github has (had) some of the worst wording in the permission prompts as well as too many 3rd party apps asking for blanket permission to do anything and everything in all your repos. It might as well be "hey, give us root on your servers and our new service will do cool things for you!"

And even worse is the 1000s and 1000s of repos that have signed up for those services including all kinds of popular libraries. They've given the keys to hack the libraries at will to 10s or 100s of random companies that integrate with github. It's insane.

> If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.

This. Limiting OAuth app access to specific repositories. I really wish this was a thing. For this exact reason I'm creating a new GitHub account for every new integration and share the repo with this account. It's technically against the TOS (they only allow single bot account per user) but there is no other way if you want security.

Maybe GitHub will implement this feature now. My hopes are high.

Aren't OAuth tokens scoped to org? Wouldn't it be easier to create an org per repo? I guess it might get expensive depending on your usage/collaborators.

Edit: nevermind OAuth tokens are not org-scoped, only apps are

> If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.

Just noting that gitlab has supported per repo tokens for ages. Yes, it's really useful, as this incident can attest.

Github already has per-repo scope, and has supported that for at least several years AFAIR.

Unfortunately no. GitHub Apps can be scoped to a repo, GitHub OAuth tokens cannot. A lot of things fall in the second category.

You’ll also discover this if you try to create a personal access token, it’s all repos, you can’t limit them to a particular repo.

Disturbingly, their own products like azure dev portal largely push you into such a grant-universal-access-for-every-contributor path

Their sec org is impressive, so these occasional massive things have been surprising

How _lucky_ that github itself was the subject of the attack in npm.

Unless I'm missing something, this attack could have gone unnoticed for a long time (it would be hard for someone to connect a random breach in their infrastructure to an oauth intrusion affecting two of their service providers).

There's more detail in this blog post from GitHub: https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

Ok, we'll change to that from https://status.heroku.com/incidents/2413 above. Thanks!

Thanks Dang. Can you also update the title? It's not only Salesforce private repos that may have been exposed.

I've changed the title to what the new linked article says (shortened to fit HN's 80 char limit).

(Submitted title was "Some Salesforce private GitHub repos stolen")

Perfect, thanks!

Makes sense. Thanks!

Neither Heroku nor GitHub are addressing the key part of this - if Heroku's Dashboard apps were compromised, and thus access was given to connected GitHub repos for download - were the secrets in the Heroku config vars for the app also visible? That is the nightmare scenario for the affected app's owners.

Heroku/SalesForce updated their status on https://status.heroku.com/incidents/2413:

"Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers' OAuth tokens, we will immediately take appropriate actions."

Sounds like they simply don't know yet how the actor got access and what else was exposed.

Looks like it's addressed in an update to https://status.heroku.com/incidents/2413

The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts

While the situation can get worse, this token alone may not be enough...

The token isn’t nearly as concerning as what it implies: this stolen token would have been stored in heroku’s DB. However it got out likely has broader implications.

Just going to say that this is the kind of scenario I was discussing in https://mjg59.dreamwidth.org/59353.html . RFC 8705 actually provides a mechanism for dealing with this, so I guess I'm going to be trying to implement that for our infrastructure in the near future.

“A lot of services will verify the user, and then issue an oauth token that'll expire some time around the heat death of the universe”

mTLS is definitely an excellent step towards solving this.

Convincing orgs to use mtls can be difficult although perhaps incidents like this one will generate more interest.

For people looking for the HN discussion on the above-linked blog post: https://news.ycombinator.com/item?id=30916192

> Based on subsequent analysis, we believe this API key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications described above.

Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Seems like a bad idea, though I’m sure it’s hard to get off that practice if you’ve been on it for a long time.

specifically the attack Github describes seems to go:

            ???
             ↓
    Heroku/Travis OAuth token
             ↓
      Private GitHub repo
             ↓
    S3 token for NPM prod infra

Is it possible they used Github Actions and the attacker was able to access the secrets?

> Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub?

Large organisation, hundreds of developers. Kind of likely that something is visible somewhere. Still a bit surprising it’s the prod access keys for npm…

GitHub’s own Application Security product has secret scanning, allowing you an overview of secrets exposed across your entire organization.

We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.

Just wanted to check this for my own repos but it’s only available for GitHub Pro users.

For all the complexity of OAuth, an OAuth token is essentially a free pass for $stuff.

Perhaps we could start validating that a token not only is valid, but also is digitally, freshly signed with a known certificate?

I'd say there's a huge difference between:

* I stole a token

* I stole the token AND am able to keep signing said token with my victim's public certificate

A token can be stolen from many places and is comparatively easy to obtain, while full intrusion into an org's infrastructure is less likely, and even if it happens, eventually will be remediated (and the related certificate will be revoked).

Edit: looks like https://tools.ietf.org/id/draft-ietf-oauth-mtls-09.html would fit the bill.

I like this idea makes tons of sense… not only is the token valid but it’s from a trusted source… it is not going to prevent future attacks but it will add one more lock for an attacker to break before gaining access that is a good thing…

mtls is problematic because it is not proxy friendly (needs special support in the frontmost proxy).

AWS and Mastodon use HTTP request signing which seems interesting.

I think HTTP request signing will be a non-starter for an interoperable standard like oauth because of the difficulty of canonicalization without an authoritative library. AWS can pull that off because they have an SDK that will canonicalize requests for you, and it's tested in lockstep with the server accepting it. Historically, XML (and JSON to a lesser extent) have been security minefields for this exact reason.

AFAIK according the draft HTTP request signing rfc [1] defines explicit instructions for normalising, serialising and signing HTTP headers (not body). However, you can use an HTTP Digest header for example, that is a hash of the plaintext body to authenticate the body

[1] https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...

Isn’t this how OAuth 1.0a work? It is more secured for a reason.

https://developer.mastercard.com/blog/why-mastercard-doesnt-...

The other competing standard draft to OAuth mTLS is DPoP: https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop

Private repos of Heroku customers, to be more specific. Including "Salesforce" in the submission title only causes confusion.

I was confused when I wrote the title because they referred to Heroku as Salesforce and I’m not used to that.

I was confused because I got an email from Salesforce to my personal email. (We use Salesforce at work.) Email looked phishy (e.g. unsecured (non-https) links). Apparently they acquired Heroku in 2010.

> Known-affected OAuth applications as of April 15, 2022:

> Heroku Dashboard (ID: 145909)

> Heroku Dashboard (ID: 628778)

> Heroku Dashboard – Preview (ID: 313468)

> Heroku Dashboard – Classic (ID: 363831)

> Travis CI (ID: 9216)

Developers at Heroku and Travis are going to have a rough Friday

This is why it's important to have 'bait' to know if secrets are stolen or attackers have access to something they shouldn't.

For example, my site, serverthiefbait.com might help...

If github themselves hadn’t been targeted then would their security team have even detected this? That is scary to me!

This wasn’t clear to me at first but it looks like these might be heroku specific repos and not salesforce ones.

Not to disagree, but to provide context, Heroku is wholly owned by Salesforce.

Yea it’s just not clear if the repos are heroku infrastructure, heroku users or something else.

So this means they also got access to all the private npm package code? Compromising all the organisations using travis or heroku is one thing. But all npm packages quite another.

Can anyone confirm or rule out whether config vars for heroku apps have been exposed?

I can't see anything mentioned on the incident page nor config vars page

https://status.heroku.com/incidents/2413

https://devcenter.heroku.com/articles/config-vars

I don't think they know yet.

It's pretty plain that source code exfiltration occurred, though. It's not clear to me how exactly to confirm that the exfiltration happened to your account or not.

From the other thread:

> GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.

Probably for a lot of us, a source code leak is not the end of the world, but leaked secrets (i.e. config vars) should be addressed immediately. Rotating credentials will take time. E.g. 20 heroku apps with 30 secrets per app could realistically take several hours (even days) to fully rotate creds.

A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents.

Hope it can help some of the organizations dealing with this right now!

https://blog.gitguardian.com/a-practical-guide-to-prioritize...

I still don't totally understand, can anyone read between the lines to explain what's going on here? The attackers stole what sort of OAuth tokens with what sort of access?

I could be wrong but you know when you sign up for services you have option of signing in with github or other oAuth providers? I think they used this to gain access to various private repos, npms, cloud credentials.

So it seems like if your company had some super secret project running on github, heroku, AWS, they had full access to it.

This is my rough understanding just skimming so could be wrong.

I guess here's what I'm wondering: did they somehow get multiple Oauth keys, one per account compromised on github -- from both Heroku and Travis, just coincidentally both heroku and travis exposed their customers keys somehow?

Or did they get one Oauth key each from Heroku and Travis that exposed all of their customers data on github? Are there keys that would do this, set up for what purpose?

Am I right to say GitHub knew about this on the 12 April [1], but Heroku customers are only finding out now, 4 days (!!!!) later?

Heroku boasts:

> we will notify affected customers by email without undue delay.

4 days. Really. "without undue delay".

[1] https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

> Following immediate investigation, we disclosed our findings to Heroku and Travis-CI on April 13 and 14

13-14, so "only" 1-3 days later, depending on what GitHub mean and timezones involved.

Anyone else noticing an uptick in OAuth bug/authentication integrations for CICD/dev ops platforms?

Yikes. This is bad. On a friday night no less.

Some questions that come to mind:

* Would encrypted secrets for GitHub Actions be exposed to these tokens?

* How to best audit for unauthorized access to exposed repos?

* Were Heroku encrypted secrets exposed?

Nothing can pull the value of secrets from GitHub actions without either running a compromised/modified runner, or the workflow itself encrypting/printing/shipping that actions secret off to somewhere. If you’re really worried, use oidc GitHub actions where possible: https://docs.github.com/en/actions/deployment/security-harde...

Good to know - thanks!

>On a friday night no less.

Not just any Friday, but Good Friday, which happens to be a legal holiday in many countries, including here in Germany. And since Easter Monday is one too, a lot of people will go on vacation on this "long" weekend (maybe even taking a couple of days off before or after to make it even longer)... Flights and hotel rooms booked, and all that.

Its not fully clear to me, entire private repos been confirmed stolen?

My reading is if you allowed Salesforce/Heroku or Travis CI access to your GitHub account/organization, then there is a chance that that grant was leaked to a malicious third party - and they may have accessed your github account and downloaded private material.

GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.

OAuth properties are like jet plane cockpit buttons. For computer science students there should be one semester OAuth course. And in big companies there should be OAuth Expert. That is how much knowledge OAuth requires. If you don't agree with me try using Keycloak. It has EVERYTHING so you don't need to implement.

Reply on news.ycombinator.com