So it wasn't phishing attack? Wonder how those bot access tokens got stolen.
It's an unfortunately common problem with GitHub Actions, it's easy to set things up to where any PR that's opened against your repo runs the workflows as defined in the branch. So you fork, make a malicious change to an existing workflow, and open a PR, and your code gets executed automatically.
Frankly at this point PRs from non-contributors should never run workflows, but I don't think that's the default yet.
I think the mistake was to put secrets in there and allow publishing directly from github's CI.
Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue.
https://docs.pypi.org/trusted-publishers/adding-a-publisher/
For a malicious version to be published would then require full merge which is a fairly high bar.
AWS allows similar
This incident reflects extremely poorly on PostHog because it demonstrates a lack of thought to security beyond surface level. It tells us that any dev at PostHog has access at any time to publish packages, without review (because we know that the secret to do this is accessible from plain GHA secret which can be read from any GHA run which presumably run on any internal dev's PR). The most charitable interpretation of this is that it's consciously justified by them because it reduces friction, in which case I would say that demonstrates poor judgement, a bad balance.
A casual audit would have revealed this and suggested something like restricting the secret to a specific GHA environment and requiring reviews to push to that env. Or something like that.
You can't really fault people for this.
It's literally the default settings.
Why is this a problem? The default `pull_request` trigger isn't dangerous in GitHub Actions; the issue here is specifically with `pull_request_target`. If all you want to do is have PRs run tests, you can do that with `pull_request` without any sort of credential or identity risk.
> Hilariously the people at pypi advise to use trusted publishers (publishing on pypi from github rather than local upload) as a way to avoid this issue.
There are two separate things here:
1. When we designed Trusted Publishing, one of the key observations was that people do use CI to publish, and will continue to do so because it conveys tangible benefits (mostly notably, it doesn't tie release processes to an opaque phase on a developer's machine). Given that people do use CI to publish, giving them a scheme that provides self-expiring, self-scoping credentials instead of long-lived ones is the sensible thing to do.
2. Separately, publishing from CI is probably a good thing for the median developer: developer machines are significantly more privileged than the average CI runner (in terms of access to secrets/state that a release process simply doesn't need). One of the goals behind Trusted Publishing was to ensure that people could publish from an otherwise minimal CI environment, without even needing to configure a long-lived credential for authentication.
Like with every scheme, Trusted Publishing isn't a magic bullet. But I think the proscription to use it here is essentially correct: Shai-Hulud propagates through stored credentials, and a compromised credential from a TP flow is only useful for a short period of time. In other words, Trusted Publishing would make it harder for the parties behind Shai-Hulud to group and orchestrate the kinds of compromise waves we're seeing.
I just went to github to search for references to that trigger-type, and I admit I was surprised at the sheer number of times it is visible in a code-search.
It seems like a common-pattern, sadly.
“ At 5:40PM on November 18th, now-deleted user brwjbowkevj opened a pull request against our posthog repository, including this commit. This PR changed the code of a script executed by a workflow we were running against external contributions, modifying it to send the secrets available during that script's execution to a webhook controlled by the attacker. These secrets included the Github Personal Access Token of one of our bots, which had broad repo write permissions across our organization.”