PHP's Git server compromised, moving to GitHub | Hacker News Reader

PHP's Git server compromised, moving to GitHub

news-web.php.net

405 pointsjsmith45135 comments

Zerodium[0], the magic string searched for in the fake user agent, is a zero-day acquisition platform, in case anyone else is wondering. The third parameter to `zend_eval_string` in the commit [1] is the "filename" [2] of the executed code, which is probably used for stack traces. Which is quite a strange choice, because seeing that in the logs would probably ring alarm bells immediately. Overall, this commit seems strange in its structure IMO - not in a "good" covert way, but in a "why does it not try to hide better" way.

EDIT: According to zerodiums charts, this would have been a payout of up to 250k$. However, I'm not quite sure what the attackers plan on cashing out on that would have been, given that the exploit would have quite obviously referenced zerodium and therefore would make the submitter the primary suspect of being the author. Maybe some APT used this "simple" format to disguise the actual power? But then again, it would be in their interest for the exploit to actually go through.

[0] https://www.zerodium.com/

[1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...

[2] https://bugs.php.net/bug.php?id=78224 (only mention I found with a documented use of the parameter)

[3] https://zerodium.com/program.html

Zerodium say they have nothing to do with this: https://twitter.com/cBekrar/status/1376469666084757506

>Likely, the researcher(s) who found this bug/exploit tried to sell it to many entities but none wanted to buy this crap

but why so snarky?

twitter. the takes are hotter than the sun, the style is snarkier than zkSNARKs.

am I understanding it correctly that somebody had breached the PHP git server and tried selling it to 0-day and similar exploiting businesses?

That was my assumption given that they explicitly mentioned zerodium. But so far only the commit itself is public, so until we know more about the attack or the attacker, this is pure speculation. The tweet from zerodium is not clear on whether they received an offer for this vulnerability, unfortunately.

> Zerodium say they have nothing to do with this

Would they admit if they did?

"zerodium investigated zerodium and zerodium found out that zerodium had nothing to do with this"

Well the opposite may be true too - the comment may be lying about it having anything to do with zerodium or that date.

Could be just a meme or a bad attempt of misdirection in case it went undetected for a while.

That's unknown, but it would be strange for them to be involved - it would not make much sense for them to pay out a bounty to themselves.

I think it's far more likely that someone tried to plant this there in order to cash out via zerodium.

Maybe the person that reported the exploit to Zerodium already cashed out in mid 2017 and was angry that the exploit hasn't been fixed yet. Therefore, he decided to make it very obvious and force php to react.

I believe the exploit itself was introduced with the commit in question, it wasn't there before.

The other exploit, the one that allowed the commit to be injected.

As has been pointed out elsewhere, the exploit is the ability to inject code into the PHP Git repository. Not this RCE.

The commits say it was 'sold to zerodium, mid 2017', which would lead me to believe that someone has already been paid for this.

I'd say the reason it doesn't hide better is because it's specifically meant not to hide. It's burning the vulnerability that has apparently existed for almost 4 years.

The vulnerability most people here are talking about was introduced to the PHP codebase last weekend, as part of the block of code containing the "sold... mid 2017" message -- that vuln wasn't there previously, at least in that form. It's not clear what exactly is going on here, but if something was sold to Zerodium in 2017, it wasn't that vuln.

So, the message "sold to zerodium, mid 2017" is there, but may not be true -- it might very well be deliberate misdirection.

Perhaps the vulnerability in the git server that allowed this is what was sold, rather than the commit.

I don't see much point in discussing these latest commits. It's very unlikely they were actually malicious in intent (at least to PHP).

The vulnerability that is of interest is the one that allowed these commits to be injected. That is what I figure was sold years ago, and was burned recently.

Problem is, burning the exploit this way only works if someone does the work to find it.

Of course, it could appear in the server logs (or be detailed on a file dumped on the filesystem) - but it seems PHP have just decided to move to Github, and may not actually care about this, so it may not see the light of day.

Sorry, I'm not sure what you mean. The commit was almost certainly never meant to make it in to some server log, it was meant to be seen, and it was. In that way it 'burned' (ie destroyed) the backdoor in to the PHP code repo.

And PHP does seem to care about the real exploit, the one in their infrastructure, which is why they're moving to Github.

Mind expanding on your above comment to help me understand?

Meant to be seen with a commit named "fix typo" wtf are you talking about? It's only due to luck that it has been reverted quick enough before being available through binaries.

I mean exactly what you did - "The vulnerability that is of interest is the one that allowed these commits to be injected".

Yes, it will be evident that the commits appeared in the git history, but not necessarily how they got there. If the git repo on disk suddenly has that commit, and no other usual indicators of compromise, that doesn't really give you anywhere to start looking for, what we agree is, the interesting vulnerability. All you have is "we know this box/service was definitely rooted, so look at all our infrastructure end-to-end".

They care about the exploit enough to move - of course. But do they care enough to investigate and find the actual exploit? Or hand over the server image to someone who does? Unless they've wrapped it in some PHP, it's probably not their core competency. I don't follow the PHP project, so I don't know the answer to that.

Okay, I see what you mean. I guess I assumed they wouldn't just accept that their server was hacked and not do a through investigation (or have someone do so), but I suppose that's mostly because I couldn't imagine not if it was me.

> it was meant to be seen, and it was

Are you sure about that? If anyone wanted to demonstrate their access/power, they would probably leave some way to identify themself as the author - with a half-hearted concealment like this, anyone could claim to have been the hacker.

The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.

> The only way this discovery would help the hacker, is if they intended to force a security audit of the karma system and/or the move to GitHub.

I agree. Of course I can't be sure of anything here, but to me this commit seems suggestive that there's other injected code somewhere since mid 2017.

You also make a good point on attribution. I had considered that too and if not wanting to take credit could tell us anything about the author.

For example say you were an intelligence agency that knew about this access and knew it was being used by an enemy. Perhaps you couldn't let php know about the vulnerability without exposing that you had access to other data allowing you to know about it. Creating this commit could be a way to share knowledge without it being known where/how it was found.

Of course that's all wild speculation.

> If you are not part of the organization yet, or don't have access to a repository you should have access to, contact me at nikic@php.net with your php.net and GitHub account names, as well as the permissions you're currently missing.

Migrating accounts of PHP core developers by asking them for their username over unencrypted email? What could go wrong? A security incident already occurred, so we should be good on that front for a few years, right?

Are github usernames especially sensitive?

"I'm xy and had push access to the php core git server, but I do not yet have access to the PHP organization. Could you add @xyfake to it? Thanks!"

Combine that with a spoofed/typo'd sender and you're good to go.

You can mitigate that pretty easily by checking whether recent commits from this person are linked to that GitHub account.

If the person had a GitHub account before, it could be newly created by the attacker. Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com.

In a perfect world, this would be fine, yes. But when adding work on top of a probably already very stressful workday, the likelihood of this creating additional problems is there.

> If the person had a GitHub account before, it could be newly created by the attacker

Doesn't GitHub require e-mail validation before it associates commits with that e-mail address?

> Or they could have had different e-mails for php.net and GitHub, i.e. something like php@xy.com and github@xy.com.

Then require them to add their php.net mail address to their GitHub account.

> the likelihood of this creating additional problems is there.

That I agree with.

> Doesn't GitHub require e-mail validation before it associates commits with that e-mail address?

IIRC, nope! You can say your email is Linus Torvalds’, and GitHub will not question it (and even link to his GitHub profile in the commit history!)

> You can say your email is Linus Torvalds’, and GitHub will not question it (and even link to his GitHub profile in the commit history!)

That's the other way around though -- claiming the victims GitHub profile made a commit controlled by the attacker, instead of claiming the attackers GitHub profile made a commit controlled by the victim.

No, you could claim the commits made by the victim this way. The steps would be:

1. Look for a contributor without GitHub account (i.e. xy@gmail.com )

2. Add xy@gmail.com to your GitHub account/create one with that address [0]

2.1 The commits made by the victim will now link to your account on GitHub

3. Send a mail to Nikita with your faked GitHub-profile

None of this will survive a thorough check, but under pressure this will easily pass a surface-level check.

[0] I think you need to confirm your primary E-Mail, but not secondary ones. Alternatively, you could look for commits made from an now deleted mail account.

Yup, AFAICS GitHub associates secondary addresses with your account without verification.

Although the victim would be notified that their address was added to a rogue account, the notification mail says: "If this wasn’t you, please ignore this email." Ugh.

> recent commits from this person

Given that the attacker managed to push unauthorised commits, I don’t see how recent commits can be considered a reliable method of authentication.

You need to audit the commits anyway, and presumably after the audit they can be trusted again.

Isn't there an existing, working PKI for this? "I'm xy and have been signing commits using the key 0xdeadbeef for some time now, signing this message with the same key."

(Yes, this still has holes, but it's a far smaller attack surface than "hello, it's me, trust me because I say so".)

Looking at the commit history [0], it seems that the majority of the commits aren't signed. So no, this would probably not work for identifying everyone.

[0] https://github.com/php/php-src/commits/master

For now I'm only handling people for which I recognize the account or can confirm recent commits.

I assume you mean that you are only handling people for whom you can confirm their recent php username AND can verify that the specific username connects to/is owned by the same person as the github account provided?

Just to make sure that someone doesn't go in and find a PHP account with recent commits, then email you with a third party github account.

For most contributors we know GitHub identities, as PHP was using pull requests etc already from GitHub.

Aah, k, makes sense :)

Are you requiring signed commits? If you're not, I strongly suggest that you do in the future.

I've been pointed out once that only tags should be signed - and referred to this: [1] https://news.ycombinator.com/item?id=12290873

The whole part of signing commits is providing proof that you are who you say you are with the same level of trust provided by other systems built using Public Key Crypto.

You need people to properly secure their keys of course, but it makes life a lot harder for an attacker.

I sign my work-based git commits. I've been using GPG since the late 90s. It's not a silver bullet (especially when a git server has been owned).

When you git pull, then do your work and git commit/push, if you are not careful and you do not examine git status/diff you could sign and commit the hackers change (that you pulled from the repo initially) along with your new changes. So care is required even with PGP signed commits.

This is another good reason to not interrupt developers while they are working. Doing so, makes mistakes such as this more likely.

Can you walk me a scenario where you and I are hosting our code on a malicious git server and Chuck can get their code in our our codebase? (assuming only signed commits)

See:

> if you are not careful and you do not examine git status/diff you could sign and commit the hackers change

I assume he meant that you pull, get an auto merge and then inattentively sign it. With this, you confirm the hackers change with your signature.

You could add a push hook to only allow fully signed chains, but given that we are talking about a compromised server, this does not help.

But how does the hacker's commit goes through in the first place, if all PRs with unsigned commits are rejected outright?

If he hacked the server (which was assumed in the grand parent), he can probably shut off that check rather easily

And then you have a singled commit without a signature lying there for everyone to see and call attention to you.

You don't really pull down code though, you pull down commits. I'm not seeing a case where you sign the attackers code. You might sign a merge commit that has an attack commit somewhere in the tree, but that's very different and much more easily audited.

Even if we assume the git server is completely malicious, I'm still not seeing a case where an attacker can hide a change because I didn't 'status/diff'. I'm open to considering attacks I haven't thought of though.

I think the problem on GitHub is that if you compromise someone's GitHub account (from a leaked password or whatever), you can just add a new public keypair to the account, and now you can sign code as that user.

Requiring two-factor auth for the GitHub account should be good for this though.

The notice starts out by saying:

> Yesterday (2021-03-28) two malicious commits were pushed to the php-src repo [1] from the names of Rasmus Lerdorf and myself.

…and ends by saying:

> We're reviewing the repositories for any corruption beyond the two referenced commits. Please contact security@php.net if you notice anything.

Are you sure that using recent commits as confirmation is a good idea?

There will be increased scrutiny on this repo anyway. It's not necessarily the safest, but probably a low-risk and easy way to migrate committers.

Can you provide more details about the breach? Did they get in via munin?

Trying to sneak that diff in was a hugely wasted opportunity, it sticks out like a sore thumb. If they were going to commit anything, it should have looked like a legitimate change. Another option might have been to wait until just before a release and fiddle with the tags or the newly opened QA branch (if any).

Check out https://freedom-to-tinker.com/2013/10/09/the-linux-backdoor-... for another attempt, that one at least requires careful review to notice the problem

They were presumably testing the waters to see if they were noticed first.

It does beg the question if this was only the first time they were noticed.

but why would you test that so obviously ('fix typo' but add code)? and they made it clear that they likely had access to the git server (by pushing 'from' different accounts), which (obviously, if that's what is actually happening) can kill their access

In the diff, could they be hinting that they've had access since 2017? [1] Maybe they just wanted to burn it now for some reason.

[1] https://github.com/php/php-src/commit/c730aa26bd52829a49f2ad...

It does make me wonder if something else was changed and this was what they wanted you to notice (as a diversion).

I think that's a good point. Apparently they had the ability to make commits as any user, which was a huge opportunity. Then they waste it on such an obvious backdoor? It does seem possible that this was a diversion tactic.

Edit: Elsewhere someone mentioned, this could have been "marketing" - to demonstrate their ability to take over accounts.

> this could have been "marketing" - to demonstrate their ability to take over accounts.

Hot take - They burned it, in a very visible way, to prove they've had access since 2017 (or whatever the investigation will reveal). A full security audit isnt very feasible.

Would force everyone to either downgrade and face known vulnerabilities, or do nothing and face high-risk unknown vulnerabilities.

I don't see any way they could have snuck anything in unnoticed as a large patch. The number of people who regularly do big patches to PHP is not that large. If you send a big commit as one of them, that person would likely notice and raise an alarm they didn't do it. If you send as somebody who doesn't regularly make big changes, then people would wonder how come this person, who never submitted anything big, suddenly submits a large change without any discussion. So it would have stuck out anyway.

It could work if you do it in some poorly maintained extension, maybe, but that extension probably wouldn't be much used either.

Committing to release branch would be the worst way of doing it, since only RMs commit there, and RMs are those people who are going to notice if somebody commits to their branch and it's not them. Also, since releases are tagged manually, it'd not get into the release anyway, unless you somehow trick the RM into merging the change into their local repo without looking. Which they have no reason to since they created the branch and only they are supposed to commit there. So it'd require some serious trickery.

Note: I did not keep the original title (Changes to Git commit workflow), because it sounds mundane, rather than being related to an apparent compromise of the git server of a high profile open source project.

Run your own infrastructure. It will be fun, they said.

HaHa I think some point like the challenge until...

Moving to github doesn't change anything fundamentally. It just moves infrastructure hosting to someone else (even though it may or may not be a more secure hosting).

Something like signing commits, and requiring all commits to be signed properly with a known key (and checking this on push and before release) would be an independent verification layer, and a more fundamental response to the compromise.

It moves infrastructure to someone else _whose core job it is to manage this infrastructure_. I definitely trust GH more than myself in managing a git infrastructure securely

That's assuming the self-hosted git server itself was compromised, and it wasn't just a simple matter of the attacker stealing some committer's private keys.

If that's what happened, then moving to Github solves nothing.

> I definitely trust GH more than myself in managing a git infrastructure securely

That's FUD, and appropriate too considering who owns Github now.

in comments on the first commit it was said that the access was revoked, but another one appeared, which points at the server

Did you notice that the commit was reverted, but then almost immediately re-applied? This is a strong indication that the server admins did not know the how the server was compromised.

With Github, you can: (1) use "events" API to see which account did the push and (2) remove the account's ssh keys and change password from a secure machine to stop compromise right away. This stops the compromise completely.

Compared to that, if there was a compromise on our own server, it is quite hard to tell that everything was cleaned. Remember kernel.org compromise in 2011 [0]? They took a few weeks to recover, and this was with some of the best programmers.

[0] https://lwn.net/Articles/464233/

It's the style of the times: techno-feudalism. Serve a powerful enough feudal lord by doing your business on their servers and you'll be protected.

While it's not clear yet whether it was pwned infra or leaked privkey moving to GitHub would definitely lower chances for the former. When SDL moved to github recently I brought this up as one of the advantages when the typical "but M$ suxx" comments started pouring in.

After learning Django for a few months I decided to switch back to php/wordpress yesterday, php7 and php8 look solid and php remains to be the best language for the web in my opinion, Php8 added JIT which might extend itself outside of web applications. Hope moving to github will free the team to do even more on Php development.

I wonder if GitHub will ever support requiring signed pushes. It supports requiring signed commits, but that isn't always feasible, since you could want to merge commits signed by an external contributor without rebasing those commits in order to sign them yourself.

That does seem like a really good idea. If I make a protected branch that requires signed commits I really only care the signatures on the commits you'd see with `git log --first-parent`, as those are the ones by people I trust.

I wish git would support signing with tools other than GPG. I hate using it and tools like minisign/signify are a huge improvement.

Hell, the entire thing is simple enough to just be integrated in git.

It would have been interesting if GitHub had purchased Keybase instead of Zoom. I'm still a bit bitter over that sell out.

My PHP is poor, but it looks like it was being changed to check for a string coming in via a "typo'd" HTTP_USER_AGENTT header, that started with zerodium, and then eval the string without the zerodium prefix on the server.

Reminds me of that Linux compromise years ago where they picked it up because the change was directly on the build box, except at least that one was hidden behind an apparent `=` vs `==` "bug" - this one has a bit less style.

It is C not PHP in the linked diff :)

Its not actual PHP code, it's C, which PHP is written in

Ah - that explains a lot, I was expecting to understand a lot less of it ;)

Maybe this was someone close to Zerodium burning the compromise against PHPs Git server to bring attention to other zerodium commits dating back to 2017?

Zerodium buys vulnerabilities and exploits, not direct access though. You can’t sell them this kind of backdoor.

I mean with keys to the kingdom, you can just plant backdoors and such every now and then and sell it off on Zerodium et al.

Poking around on https://git.php.net, it appears to be using "gitweb":

  <!-- git web interface version 2.11.0, (C) 2005-2006, Kay Sievers <kay.sievers@vrfy.org>, Christian Gierke -->

"'Gitweb' is a Git web interface. It is written in Perl and can be used as a CGI script, or as a mod_perl legacy script (run by ModPerl::Registry handler). It allows browsing a git repository (or a set of git repositories) using a web browser." - https://git.wiki.kernel.org/index.php/Gitweb

The last commit looks to be Dec 2020: https://github.com/git/git/tree/master/gitweb

Though I don't know that this was how they got into the git server.

Disclaimer: wild speculation below

There's a strong possibility the vulnerability lay in their hosting setup rather than in the software application itself.

There's an argument to be made that self-hosted software that are non-trivial to configure for security have "dev ux" type security bugs, but I doubt that'd constitute a CVE. It could also be unrelated to the git-web hosting config.

Looks like Softlayer / IBM Cloud. There are some high numbered ports listening and accepting connections, including git/9418, which is interesting.

Was PHP's Git server using a self hosted Gitlab instance? I can't find the past threads, but I thought there was an announcement a while back that they were moving to Gitlab self-hosted. Maybe I'm misrembering which project that was?

It seems they use some combination of gitolite and their own stuff.

The post mentions karma, there's some code here: https://github.com/php/karma/

Thanks. That's a bit of a relief, given that we use self-hosted GitLab instances.

Well it's still vulnerable. At least in Gitlab you can use fancy features like requiring 2FA for members of a group. But you still need to secure its infrastructure.

I think the most common attacks on Github repos and groups have been on individual accounts. Which 2FA would mitigate.

> Which 2FA would mitigate.

Or Rust, right? </sarcasm> https://github.com/php/karma/search?q=exec

This looks like the work of either a talented but neophyte black hat, or an intentional burning of an exploit.

commits history have following:

  - 8d743d5  Revert "Revert "Revert "[skip-ci] Fix typo"""
  - 2b0f239          Revert "Revert "[skip-ci] Fix typo""
  - 046827a                  Revert "[skip-ci] Fix typo"
  - c730aa2                          [skip-ci] Fix typo

There is a reason why some services are worth it... for one you don't have to worry about all those issues that could pop up. It's someone else's problem.

Feel for the maintainers, tough to keep own stuff together -- but also first reaction is that another massively influential part of the web/opensource being stored on GitHub. I'm not anti-MS by any means but just reminded of the dominance of so much of our global codestores being on there

It's git. That part is easy to migrate. Exporting issues and discussions might take a bit of work, but if/when GitHub starts going downhill there'll be a script to migrate to GitLab (or whatever will be the preferred option by then)

(Yes, there's already such a script for GitLab.)

Unless the first step in MS Github's downhill journey will be to cut off APIs needed for those exports.

I wonder if the signed features of GitHub would have helped here. Probably can't ask all the 3rd party contributors to setup signing commits, but it would help to spot commits not signed.

If the attackers would have used their own name/email linked to a GitHub account and signed the backdoor commits, GitHub would display a green "Signed" label, and everything would look OK to an outsider.

That's why it's just a gimmick. Signing only works as protection against hosting compromise if the signing/verification is separate from the hosting itself.

Someone responsible for release would have to manually keep a list of keys of authorized comiters and check the repository against this list at the very least prior to a release.

The number of people with direct commit access is almost certainly less than the number of commits in a release though. Without signatures you have to verify every commit.

I'm not against signatures. I just think relying on third party system to verify them is a gimmick.

It's not about the green label. A set if commits signed by some unknown person is a lot easier to spot and clean up.

Looking at these commits they say mid-2017. Without signatures any previous commit could be by the same author.

> I wonder if the signed features of GitHub

Uh, you mean the features of ... Git?

Quick migration of toolchain. Curious how that will work out. If it does, my deepest respect to their engineering.

PHP already uses GitHub and pull requests there, it's just not been the canonical git repository until now. It's not really a big change for most; if anything it's easier.

This was so dumb. Even mentioning zerodium and skip-ci and sloppy commit message.

This was not dumb, most probably this was marketing.

They chose to burn -very publicly- the two accounts they had access to in order to showcase that they have an in that they can use somewhere else.

Dunno, they just provoked a hardening and possible review of the remaining infrastructure.

It seems likely it was meant to be seen and fixed.

Bold move

Perfect time to get your php commit bit now. Just ask:)

Reply on news.ycombinator.com