Plans to Re-enable the GitHub Integration
blog.heroku.com
blog.heroku.com
> I'd love to hear from someone at GitHub (anonymously or not) what they've done to be satisfied with action Heroku have taken that would allow the integration to be turned back on. My confidence in Heroku to give me accurate information on this is low.
As far as I can tell from Heroku's communications they:
- Have no idea how the attacker gained access
- Have no idea if the attacker still has access
If they do know these things then I've not seen them say so.
> In an effort to improve the security model of the integration, we are exploring additional enhancements in partnership with GitHub…
Github permissions possibilities continually confuse me, but integrations are always asking for more github permissions than I really want to give them, more than it seems like they should need for the integration; I'm never clear in an individual case if this is because they are doing it wrong, or because github doesn't offer granular enough permissions. Some vendors with integrations in the past, when I've complained, have _claimed_ it's because github does not offer any more granular permission that includes what they need.
This announcement still leaves it unclear which it was in this case.
I wonder if the fallout of this thing will result in github fixing whatever it is about their permissions system that is leading to integrations asking for and getting more permissions than should be required?
I have seen most blame over this kerfuffle focused on heroku, but I suspect github's too blunt integration permissions could use some ire, which might help motivate Microsoft/github to improve things.
- [1] https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-appsGitHub apps are indeed noticeably better. But that doesn’t always help
Furthermore, we basically only ask for the one “mandatory” permission - there are scores of perms you could request when authorizing an app - and that’s just read only access to the code.
read:org
(need this to list all repos of an org user is in/has created), I need the- admin:org
scope which gives me access to, "fully manage the organization and its teams, projects, and memberships."So yes, definitely not fine-grained permissions I'd say. Useful in recklessly adding more features just because you have access to more data tho haha.
This is not exactly related to the topic but in the course of this wider fiasco, we actually uncovered a bug in Github's audit logging.
The Travis CI app was removed at an org level by a Travis employee in the Middle East and to my knowledge, this wasn't publicised in advance so at first glance, it seemed kind of concerned.
Anyway, that org level event didn't actually propogate up to the enterprise/umbrella level. That is, you can have an umbrella consisting of multiple Github orgs and the audit logs are supposed to roll up into the umbrella audit log.
Anyway, we got confirmation a couple of days ago that it should be fixed now but worth a note if you used Github audit logs to respond to the Heroku incident or the Travis CI one
My armchair guess is whatever method someone used to gain access more than likely took an architectural change to fix.
As a customer, I'd like to know how the attacker got in and how Heroku can guarantee they don't still have access through those means.
I wonder what the real root cause is, organizationally? Is it corporate apathy as Heroku was swallowed up by Salesforce, and GitHub by Microsoft, or something else? I wish I had a bird's eye view inside the org. For now though, I guess all I can do is move the workload to AWS.
(We are.. a tiny fraction of Heroku, in terms of anything you like - it's excusable IMO that it was an untested procedure not smooth etc., small team with MVPs to ship.)
In 957h I would think you can start to think about bringing on a specialist on contract (or implement the new GH App based still-future version mentioned) if the permanent team can't figure it out / don't have capacity! It's not good for reputation, surely, I have to imagine it was considered low priority rather than something they actively tried but failed to fix for so long, but I don't think that's a good look, even if metrics show it's little-used or only by free tier or whatever.
I just re-enabled the Github connect, re-enabled Automatic Deploy, and removed the `git push heroku` line. 1min.
While Heroku was out I took a look at Fly.io and Render.com but neither one quite fits the same use case.
Fly does not have automated PR builds and Render does have them but they all use the same database instance and you can’t have more than one active free database (so multiple PRs affecting the database are not possible).
I actually switched to Render before realizing that limitation. It was easy to do and works well but now that Heroku has their integration working again I’m thinking I’m just going to switch back…
Setting up a well organized PR to preview environment deployment pipeline, that can be automated by CI passing is time consuming and full of yak shaving opportunities.
Just clicking “enable” for a small team is a huge win.