Heroku Security Notification | Hacker News Reader

Heroku Security Notification

status.heroku.com

413 pointschizhik-pyzhik116 comments

Open articleView on HN

This is a much bigger problem than Heroku.

There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).

- Productboard

- Bugsnag

- Sentry

- Skylight

- Percy

- CodeTree

- Databox

There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.

Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).

Solution:

- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs.

- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.

> There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access).

Sentry does not request write access to source code. It requests read/write access to issues and read access to source code. You can also see this on the documentation for the GitHub enterprise integration which lists the exact permissions required: https://docs.sentry.io/product/integrations/source-code-mgmt...

Thanks for clarifying.

It would still be better if I didn't have to give read access to source code, but could still use the issues integration. But I agree it's not as a bad as write access to source code.

Most error monitoring services want this as they map errors to commits.

As a user it's nice to have the option of foregoing the feature.

Then just don’t link it to GitHub if you don’t want your errors tracked back to specific commits?

I might want it linked to my issues but not my source code.

I know it's more work, but they could also use Github Actions to push recent commits (past 90 days) to themselves, and use them. That way they wouldn't store or have access to all source code.

For a while now I've been worried about this -- either integrations asking for full read/write access for a service that might seem to need only read.

Or maybe worse, integrations asking for access to anything my account has access to, when I only want to grant it to one repo or organization, or only to public repos and not private ones.

Whenever I've reached out to inquire/complain about this, I've been told that github does not give them granular enough auth settings to ask for less than this.

Is this true? I don't know. When I've tried looking at the relevant github docs myself, i quickly get confused.

Does anyone understand the github auth architecture -- does it need to be fixed to allow more granular access, or are integrations just not using it properly? Like... who should I be complaining to?

I've been looking into this as I'm building similar per resource scoping for conjure.so's API.

GitHub has 'OAuth Apps' and 'GitHub Apps' [0]. The former's scopes do not permit such granularity (eg the `repo` scope gives access to all repos of the account [1]).

The latter is much more granular, allowing the user to select specific repos to grant permission [2]. The 'GitHub App' owner can see their installations and also determine if the user chose access for all repos or on a per repo basis.

Netlify does such granular installation and will prompt you if you don't see your repo listed in their dashboard to check permissions.

[0] https://docs.github.com/en/developers/apps/getting-started-w...

[1] https://docs.github.com/en/developers/apps/building-oauth-ap...

[2] https://docs.github.com/en/developers/apps/managing-github-a...

Ironically, Slack seems to have really homed in on granularity of integration access as a key feature as well.

Which makes sense... but then so many companies don't do that.

Do you think that still sounds much more confusing than it needs to be?

I don't totally understand why they offer these two mechanisms of integration -- or why the granularity of access would need to be different between them. Why not let "OAuth apps" be authorized per-repo -- if you are going to have "OAuth apps" existing as a thing still? And they don't seem to be deprecated in favor of newer "github apps", right?

Yeah agreed its pretty confusing and took me some time to reconcile!

I imagine for legacy support or business reasons they maintain the OAuth API, but I too wish they add granular repo support to the OAuth API and existing tokens were migrated to an all repos scope, while new tokens were encouraged to be per repo scoping.

> Additionally, we recommend disconnecting Heroku from your GitHub repositories.

I have respect for the Heroku/Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards.

My thoughts are with the team working on responding to this incident on Easter Friday.

From what I can tell from the update from a couple minutes ago, this recommendation wasn’t deemed to be enough. Now they’re revoking all tokens.

For anyone wondering where this is....it is in your project -> "Deploy" -> "Deployment method" -> "Connect to Github".

For some reason for the single app I had used this for, there was no "Disconnect" button - so I just did the nuke option of deleting the connection right from the main settings page: https://dashboard.heroku.com/account/applications#third-part...

For checking your Github audit logs, you can go directly here (replace ORG_NAME with your own): https://github.com/organizations/<ORG_NAME>/settings/audit-l...

Or from: Organization > Settings > Archive > Logs > Audit Log

I hope we get some more clarity on the extent of this incident soon. We'll rotate our keys anyway but I really hope the attackers did not have access to the ENV vars that are commonly set on Heroku directly.

For many accounts the audit log isn’t going to show some crucial information—git activity. Clone events are only available through the REST API for Enterprise customers.

The GitHub pipeline integration doesn't show up on the third party applications page here: https://dashboard.heroku.com/account/applications#third-part...

I had to remove them one by one.

I can confirm that ~2h ago my integration was working but now it's not, so it seems they automatically disconnected everyone. Not sure if "they" is Github or Heroku here, but my master branch is not autodeploying anymore and trying to reconnect results into an error.

My personal account audit log says “salesforce-revocation-bot” removed my Heroku Dashboard OAuth Application a few hours ago.

"Github System" removed the Heroku Dashboard key and a personal user account of someone that seems to work at neither GitHub nor Travis removed my Travis CI key (????)

If you saw the user 'shairyar', he works at Travis based on linkedin profile page (but not based on github profile page). I was confused as well when I saw that.

I guess maybe, but this is incredibly badly communicated. Just now they showed up as a member of travis GitHub orgs, but the only member? https://github.com/orgs/travis-ci/people

It's very possible (likely) this is all fine, but done in a way that feels strange/fishy. I haven't even gotten an email from travis. They must be in fire-extinguishing mode.

I would have expected that they would have used a known human rather than look like someone with unauthorised access was doing so.

I see he now has access to three travis-ci organisations on GitHub.

Internet applications have become dangerously correlated.

We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.

Might be a hint if you really have not touched them at all in years. In other cases this might also stem from CI tools like GH actions.

Are you on an enterprise plan or a personal one? I know personal plans have limited logs, but I'd hope the enterprise ones would show clones (in reality, they both probably should). Kind of defeats the purpose of an audit log if it doesn't.

We're on the Team plan

For anyone not on a pro plan: I believe you can upgrade and still see the past two weeks of data. I cloned a few of my private repos last night to see how that affects my security logs and no logs appeared. I later upgraded to pro and visited /<username>/<repo>/graphs/traffic and can see the clone counts from before I upgraded. I also can see visitor counts from about a week ago. These clones still don't appear in the security logs though.

I tried upgrading my org to a Team to check traffic, but the upgrade seemingly did nothing. I do have sponsorships, so maybe it’s waiting until my next billing cycle? Can’t figure it out.

Ughh here I was at 1:30AM after a hard week, checking the news one last time before going to bed, and see this. I hate this dang industry and regret ever becoming a tech lead. I know the Heroku engineers have it worse and all, but just venting.

That’s rough. Stay strong, friend.

Probably just Heroku engineer, singular.

What makes you think that?

News on HN are usually not relevant to my work and I actively avoid reading anything work related on my off time. I’d find some other activity to relax and wind down if reading HN has an impact on you. Burnout is no joke - you need yourself rested for yourself and your team. Take care of yourself.

But why do you care? Not your company, not your problem.

I'm not a security expert, but if you're reading this and wondering what to do, a good start could be to just assume your repo was accessed, and so to run a tool like gitleaks against your repo. If it detects anything sensitive, I'd see about revoking/deleting those secrets right away.

In general, it's good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don't want your secrets to be as well). So it'd also be good practice to add something like gitleaks into your CI/CD pipeline for the future.

It's such a cryptic call to action. When people reading an incident disclosure end up wondering what to do, you're not finished writing. I haven't logged into or even used Heroku in close to a decade. Am I affected? I just toyed around with a hobby project on Heroku those years ago and didn't release anything. Am I affected? If that project didn't connect Heroku with GitHub, am I affected? It's been so long, I don't even know how to use Heroku anymore or even access my repos there. Yet, I got this message in my E-mail so that would imply that maybe I'm affected and need to do... something? Who knows, the E-mail doesn't say what to do either.

Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing.

> Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing.

Yet that isn’t even close to the best course of action. If your GitHub account was compromised and you had secrets in private repos, deleting everything in your Heroku account isn’t going to do anything to help.

A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents.

Hope it can help some of the organizations dealing with this right now!

https://blog.gitguardian.com/a-practical-guide-to-prioritize...

I do remember hooking up Heroku to Github for auto-deployments and thinking to myself something along the lines of, "why does Heroku need ALL of this access?"

It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!.. in which case I'd much rather have and setup that granular detail than have a token that goes across all my public/private repos...

Edit: I do see in my Github's integration page that the Heroku connection was used within the past week... but it doesn't show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?

> It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!

They totally do. Shopify's Github integration works this way, and it is fantastic!

These are known as GitHub apps and were designed to address this problem. https://docs.github.com/en/developers/apps/getting-started-w...

Thanks and I'm not surprised, it's a pretty intuitive feature! So really the only thing Heroku gets from these all-inclusive tokens is something to drive their type-a-head input on their integration page, right? Totally not worth it, I'd rather use Github's prompt.

Disappointed that the GitHub security log doesn’t show access for personal accounts. Would be rather nice if they temporarily made that available for a short period of time so we can see if any of our repositories have been cloned/downloaded.

I've been disappointed with GitHub's auditing capabilities.

I noticed that some members of my team were committing changes to our enterprise repositories with poorly-configured clients, so it was impossible to tell who made the commit or performed the push.

I understand that git itself doesn't have any protections against this, but GitHub knows who pushed it; why isn't that metadata available?

GitHub Enterprise has push logs.

https://docs.github.com/en/enterprise-server@3.2/admin/monit...

Do you know whether that's available via GitHub proper? I'd like the user identity for commits to be accessible through the API, but last I checked it wasn't.

(Updated for clarification: user identity is available via the API, but only the identity embedded in git itself, not the GitHub authenticated user.)

FYI looks like the stolen Heroku OAuth tokens are already being used:

https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

Both heroku and travisci were affected? I wonder if and what the common vector would be for two major providers to lose their OAuth tokens.

- Heroku might be using Travis? - They were compromise independently but the hackers decided to start leveraging them at the same time to get the most out of them? - The obvious common denominator: Github? It doesn't make a lot of sense though.

Interesting I had an OLD project associated with a Gmail email address that would send emails in some cases.

Yesterday I got a notification that someone tried logging into that Gmail account. The password was hard coded in the code…

I too have seen something very similar to this. I am extremely nervous all my private repos got compromised. Watching this very closely.

I’m assuming they have been.

This email was a one off that outside this code exists nowhere else.

The attacker got _write_ access to all of these repositories as well? That's extremely worrying. I hope github or someone will be able to track down if any code changes were made.

A forced push could introduce vulnerability into code in a years old commit that is hard to detect. I guess it will look dodgy when your next push is rejected.

It would not at all be "hard to detect", the SHAs of it and all subsequent commits would change, it would appear in the GitHub audit log, etc etc

Honestly should be pretty easy to just compare remote to someone's recent local copy and just spot check any new commits. Definitely a time sink, though

Yeah it's not the hardest thing to check for, the real problem is that you have to check and you have to know what you're seeing.

Someone who misses the memo on this one, isn't so good at git, and/or is on a smaller project could really be bad news.

If I want to revoke all Heroku's access to Github, is it the "Heroku Dashboard" I'm looking for under "Authrorized Oauth apps"? I revoked that one, but not sure if that is everything.

This is a chance to reiterate best practices:

Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.

In this case, can we confirm that Heroku environment variables were not accessed? Because if they were, even not storing secrets in the source code wouldn’t have prevented a breach.

If Heroku could confirm environment variables were safe I’d have a much better sleep tonight.

The announcement claims that, as far as they know, the attacker can't access your Heroku account, only your GitHub repo (because they only got the GitHub-issued token). So this would imply that your environment variables are safe. That being said, it sounds like they don't yet know how the attacker managed to do what they did, which means they can't rule out the possibility that more data was stolen than is currently known. But it sounds like probably not.

>The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts. With the access to customer OAuth tokens, the threat actor may have read and write access to customer GitHub repositories connected to Heroku.

Based on the above information, my assumption is that the attacker gained access to code repos hosted on GitHub, but not access to live dynos or the Heroku dashboard (that happily shows ENV vars). We'll see if this information updates as the investigation progresses, though.

However, given 1) write access to a github repo, and 2) auto-deployments from github to production dynos (if enabled), an attacker could exfiltrate env vars (among many other nasty things). However, this would trigger events in your app's activity log (new commits + deploys) and should be quick to verify that it didn't happen.

at work thats fine, we use vault or secrets manager, but neither of those are really suitable for self hosting for toy apps. What do you tell someone using github's free tier and aws/gcp's free tier?

You can store them in the repo and encrypt them with Mozilla sops. You can use a cloud managed key like AWS KMS or a GPG key.

If you manage your own key, you can store it in a password manager or use a USB hardware key to store it

You could also use object/blob storage or your local filesystem to store a config file and optionally apply encryption to that

You've just moved the problem from storing the secret to storing the key in that case though. For AWS KMS, you're now paying [0] for the key storage, and if you use a GPG key, you still need to get it into the application somehow.

[0] https://aws.amazon.com/kms/pricing/

If you're using a GPG key, you don't necessarily need it to leave your PC. You just need to "re-deploy" the secrets manually when they change from a place that has access to the key

Most Heroku users are using environment variables for their secrets.

Related HN article: https://news.ycombinator.com/item?id=31046791

This is as good a time as any to ask about Render.

I've been eyeing it recently and I'm thinking about launching my next project with it. Does anyone have any takeaways from using Render vs Heroku?

I’ve been using Render since January, after switching from Heroku for a project of mine with about 10,000 direct users at present. The project is a Rails app that imports hundreds of thousands of rows of data from XLSX files and spits out PDFs on demand from those imported spreadsheets.

I was worried that Heroku would end up costing me a small fortune as demand scaled. Plus the platform seemed to have stagnated.

I contemplated switching to AWS, but didn’t want to deal with the extra hassle of it. By chance, I saw someone mention Render on here, checked it out, and couldn’t be happier.

It’s a bit harder to get up and running with Render than Heroku, but orders of magnitude easier than with AWS. And once you’re operational, it’s a cinch.

And way, way, way cheaper.

Render is really not a very good name from a SEO point of view. Took me three or four searches to find it.

Google: "render hosting"

Id also recommend a look at Porter.run

It’s alright but things are still flaky given they are smaller and newer. It hung on spinning up a postgres instance. Builds are slower as well.

(Render founder) Were you trying the free tier? Builds are significantly faster on paid plans. I'd certainly appreciate more context: email in profile.

Hi -- yes this is on the free plan for clarification of the original comment. Good to know. I was testing out the free plans until I upgraded. https://render.com/docs/build-limits I don't see this mentioned anywhere. https://render.com/pricing should maybe show a row/column on the build speed for the different tiers.

Additionally, spinning up the postgres instance was on a paid tier. I contacted support but it self-resolved eventually after I blew away the instance and rebuilt. However, it was queued up for like 8+ hours on the first pass.

Thanks. We'll add info around free plan build speeds in the dashboard and docs.

See also GitHub's post about this: https://github.blog/2022-04-15-security-alert-stolen-oauth-u...

Travis-CI was also compromised here and that may actually affect more people than the Heroku side of this.

Discussion about GitHub's post is here: https://news.ycombinator.com/item?id=31046791

I have always wondered if putting confidential info in even private git repo was a good idea, although it seems to be a common practice? I feel like that question has been answered, for me anyway.

Definitely not a good idea. Source code gets sprinkled around a lot of locations with varying security

- it's stored locally (so any developer machine access could compromise secrets)

- it's stored on the build/CI machines (that might run untrusted code in the form of dependencies--possibly from unrelated repos)

- it can end up in build artifacts

Instead of connecting to a Github repo, you can use https://github.com/heroku/heroku-builds. It allows to create a build locally and then deploy it to Heroku. From what I've read so far, this approach has not been impacted and it should still be possible to do deployments like that.

What about simply using the heroku CLI which is the default? I've used that for every heroku project, I didn't even think to connect to github. I suppose the main benefit is that it's automated.

I might be naive but how does a threat actor get access to encrypted bunch of keys.

an employe gone rogue?

I just started a Discord server: https://discord.gg/K9ecetqn Please join if you're impacted by this incident, or interested in these topics in general.

I'd like to discuss mitigations around this and similar incidents with other HN:ers:

- Knowledge sharing: resources, how-tos, tips - Discussing prevention, mitigation, etc - Moral support and venting

If there's already such a forum (I assume there is), please send me an invite :)

The only thing in my security log over at GitHub that I'm not familiar with is a handful of "repo.change_merge_setting...Blocked a merge setting on the ${my_repo} repository" entries. Googling that provides little information. Can anyone explain what that means? Should I be worried?

That log entry happens when you change the settings on a repo for what kind of merges are allowed for PRs (merge commit, squash, rebase). It’s three checkboxes under “Pull Requests” in the settings for a repo. I can’t speak to any reason a threat actor would have wanted to change those settings though.

I see, thanks. I don't ever remember touching these settings before.

Heroku users using GitHub should start rotating any secrets stored in their repos (people still do), if the OAuth has been compromised it means your repositories could have been cloned. Non-enterprise GitHub users have no detailed audit history to see if their repos were cloned/zipped.

Does anyone know what to look for in the github audit logs, exactly?

Try here: https://github.com/settings/security-log

I used here:

https://github.com/organizations/<ORG_NAME>/settings/audit-l...

... but the real question is what would malicious activity look like, exactly?

I see a heap of "downloaded a zip of repository" but I suspect that's Heroku CI or other CI tool running.

Yeah I see lots of the same. They seem to correspond with Heroku deploys? Anyone know if that happens when a valid heroku deploy occurs? Do they download a zip?

I've reached out to Heroku support to ask.

Please let us know if you get any info from them! :)

Normal Heroku usage doesn’t download a zip because it uses git directly, but I’ve seen plenty of CI tools download zips.

I thought as much. Maybe it is CircleCI then.

Including Heroku CI? We don't use an external CI tool (or really any other integrations except GH), but I do see these download logs.

For what it's worth, elsewhere in this comment section someone posted that Github Support says the zip downloads weren't related to this incident. Reading between the lines, the compromised repos were probably accessed using normal git clone actions.

That was me who posted that :) seems unrelated, but still hoping to get that figured out anyway.

Same. Most belong to 1 member of our team, but a few belong to others on the team. They started happening about 6 months ago (unless logs only go back 6 mo). Would really like to figure out what these are. I asked GH, they said it’s not involved with this breach, and haven’t yet answered my next question (who/what is it?)

I believe OP was asking what to look out for within the security log, not how to look at the security log.

Don’t think these logs mention the oauth application being used to access at all.

Last year when I looked into this there was no automated way to get this info; the normal audit log mentioned below does not contain any info about actions from oauth-ed applications afaik.

If you email GitHub support they can pull out detailed logs from oauth app interactions from their internal tools.

I would expect the GH security team to have relevant queries ready by now, maybe even do some proactive queries and start alerting anyone who had suspicious activity. (But this is just how I'd do it I have no special insight if they are doing this or something else).

Tangentially, You can view your oauth linked services here https://github.com/settings/applications

To me, this reads like someone got read access to all the source code on Heroku, so if you have any credentials hardcoded in there, now's the time to change them, too.

I wonder if the hackers were kids who got bored around Easter holiday - meaning Heroku's security is shit - or if Heroku deliberately waited to announce this during Easter holiday to minimize the attention it gets - meaning they are as deceitful as all proper megacorps.

I haven't been able to trust their status page to accurately reflect what works and what doesn't for a long time. The only reliable signal is when their status page goes offline ;)

Based on their incident report, it sounds more like someone gained access to the OAuth tokens used to authenticate with linked GitHub repos, which would actually be both read _and_ write access to all source code of repos linked to Heroku apps.

RE kids or misdirection: as with all things, it's probably somewhere in the middle -- a somewhat-sophisticated attacker and a slow, evolving investigation unfortunately coinciding with a holiday weekend. They say the report was received 3 days ago (on April 13) from GitHub after they noticed suspicious activity on April 9.

Heroku Support is sh*t. Thanks. I can't access my account now.

Ah yes, the Friday night security incident drop.

Trash day

The Easter egg.

Reply on news.ycombinator.com