Canonical Ltd source code repositories have been compromised | Hacker News Reader

Canonical Ltd source code repositories have been compromised

github.com

316 pointsrvnx74 comments

So... what’s the risk? If a person updated their Ubuntu computers this or last week, are they in trouble?

That's actually the important question that Canonical should answer ASAP.

Ironically I only use Ubuntu in a sandboxed environment, fresh install every time. I run apt get update and dont do any personal info stuff on it.

This is out of sheer laziness because a USB stick is easier to plug in than Sata. Today I learned it is now has security features as well :)

Canonical is a great thing in my life. Hope they haven't been hit too hard and that they learn how to prevent it.

I do not see how this will make you safer than running the distro from a disk drive. I believe you are at a higher risk than others because you are running apt update on every boot and thats how you would get infected if launchpad was compromised - but it is not.

Probably not. The vast majority of Ubuntu development occurs on Launchpad. A few important projects are developed on Github, though. Notably, LXC and LXD. The maintainers of these projects are quite active, and would likely have seen any harmful changes being made.

The LXC and LXD repositories are under https://github.com/lxc which is a different Github organization with a much much smaller set of members (17 currently) than CanonicalLtd.

We require 2FA for all accounts under the lxc organization and only grant the access that's actually needed by those contributors. So while it's not impossible that one of our members' credentials may get compromised, especially when considering the use of access token, current access is as restricted as you would expect for those repositories.

It's also worth noting that because of Git's own design, even should one of those accounts get compromised, it would be fairly simple to spot and revert any changes that may have occurred.

Thanks for pointing this out!

I am more afraid of someone compromising the linuxcontainer.org image servers. I wonder what security they have on place...

Hijacking top comment...

We can confirm that on 2019-07-06 there was a Canonical owned account on GitHub whose credentials were compromised and used to create repositories and issues among other activities. Canonical has removed the compromised account from the Canonical organisation in GitHub and is still investigating the extent of the breach, but there is no indication at this point that any source code or PII was affected.

Furthermore, the Launchpad infrastructure where the Ubuntu distribution is built and maintained is disconnected from GitHub and there is also no indication that it has been affected.

We plan to post a public update after our investigation, audit and remediations are finished.

Thank you, your trust in Canonical is important to us, which is why we make privacy and security a priority.

-David on behalf of Canonical

Thank for the update, David. Would this be an affected store? https://us.images.linuxcontainers.org/images/ubuntu/bionic/a...

From our investigation so far, they do not appear to be, No.

At least it wasn't as juvenile as the (possible actual) juvenile "hack"[0] of Gentoo's repos that had insertions of "rm -rf /" at the wrong places (where they wouldn't even execute) and insertions of racial slurs into readmes.

[0] https://wiki.gentoo.org/wiki/Project:Infrastructure/Incident...

Launchpad doesn't look very trustworthy either: https://launchpad.net/projects/+all

Launchpad is public, think GitHub but for packages. The Canonical Launchpad repositories weren't tampered. Presumably a lot fewer people have commit access to LP since it's used for package distribution.

I just recall the takeover of the Lubuntu project by this kid, kicking out all others and his attempt to threaten others in the project at Christmas time as a ubuntu tm licensee for his website impostering as a Ubuntu trade mark owner. His forged evidence provided worked with Github staff to get others blocked. I don't trust Ubuntu accounts anymore and I don't trust Canonical because they let it happen.

While it’s troubling that this happened isn’t it sort of cool that it’s out in the open for everyone to see? At least due to open source the community can know what happened and even look for changes.

What sucks is that I just updated WSL immediately before coming to this article. Hopefully it's ok.

I only see a list of files. Can you link to something more specific?

some stranger repos have been created and there is this issue https://github.com/CanonicalLtd/CAN_GOT_HAXXD/issues/1

Removed or made private (I see a 404 error only)

Somebody on twitter took some screenshot https://twitter.com/dclauzel/status/1147525512794988544

Yes, somebody did ;)

Air France représente !

Do you mean Air Rance

Ah, I see you’re a man of culture as well

Air Lyon, tu veux dire ?

Lyon ? r/Lyon !

For sure the best of Reddit is on r/France.

We can see there the palatial fineness of french critical mind.

Actual link to screenshot: https://pbs.twimg.com/media/D-zUQ21XoAQeCgj.jpg

It shows 10 empty repositories with names CAN_GOT_HAXXD_%d and age 2 hours.

I was told this several hours ago on IRC - this is not an official statement:

someuser@somewhere.canonical.com: I can confirm that we're aware of the issue, have done some initial remediation (e.g. shutting down CI systems that might pull potentially compromised code) while we do a more in depth investigation.

someotheruser@ubuntu/member/username: And we've of course revoked the access that was abused.

We are lucky that they decided to vandalize instead of hiding something. Remember, on Github you can actually submit your commit with another’s email, and github automatically puts the identity associated with the email allowing people to spoof.

Always sign your commits!

That’s not just on github. That’s totally a feature of git itself. You can set the email address to whatever you want.

Not knowing the details around this, but if they haven't done so, Github could add something to indicate whether particular commits were signed by the PGP key Github knows to belong to the user the commit claims it was authored by. A user could have a setting to enable Github warnings where there are commits supposedly authored by user but not signed. They could also always warn where there are commits authored by a user but signed by a key the user didn't inform Github of.

Likewise, I wonder if git differentiates signatures where the keys are set as trusted in gnupg, and those that are not.

> Not knowing the details around this, but if they haven't done so, Github could add something to indicate whether particular commits were signed by the PGP key Github knows to belong to the user the commit claims it was authored by.

They do this

Github could simply reject commits with email addresses not added to your account. As you need to verify every email you add to your account this problem would be solved instantly.

and be replaced by a bunch of other problems instead. E.g. no publishing of existing repos on Github if they happen to contain a commit with a non-linked address...

One wouldn't be able to upload an existing open source repo with loads of existing contributors. That seems to really go against Github's mission of being a universal, easy-access hub for open source git repositories.

> Github could simply reject commits with email addresses not added to your account.

Sure, if GitHub wanted to completely abandon the decentralized nature of Git and be a completely centralized system. Which it would love to, I'm sure, but I don't think the user base is quite ready for them to completely shut off any repository with any commits not made by Github users, and posted to GitHub through the account of the user who made the commit, which is what you are suggesting.

This would completely break you merging others' patches into your repository outside of GitHub (with git-am(1)).

Github already shows signed commits: https://help.github.com/en/articles/signing-commits

... and today PGP was poisoned

Link, please?

Probably this: https://www.zdnet.com/article/openpgp-flooded-with-spam-by-u...

It has no effect on GitHub, though, which doesn’t use keyservers as part of its validation process.

PGP still works fine. It's SKS that doesn't, but you don't need SKS to use PGP.

If they had an SSH key only then signing would be a defense. But if the attackers got OAuth access, stolen credentials, or a personal access token to someone with access to the org, they could do everything through the web interface which is signed by a common Github key.

I trust that GitHub key less than personal keys for exactly that reason.

Commits made via the UI aren’t signed at all. There is no “common github key” for signing commits

https://github.com/aidanh010/test/commit/ebdf858ae90ab7d3ce5...

https://imgur.com/a/fSKnL9C

Isn't commit signing a bit like NuGet package signing? Not worth it unless _everyone_ does it all the time? Any valid commit doesn't necessarily invalidate unsigned commits. It only means this commit is really done by me, but it doesn't me the other commit wasn't done by me.

Yes, but there are organizations that require all commits to be signed.

When you sign git commit, you do not sign your diff but the whole source tree as you see it. Effectively, you're vetting all prior commits on the branch.

Thus maintainers usually sign merge commits and it's their job to confirm that commits don't do anything shady and come from people they claim to come from.

But github encourages merges through the website, which means they are not signed.

I wonder if it was an actual outsider attack, or just a disgruntled (ex-)employee?

> We are lucky that they decided to vandalize instead of hiding something.

We do NOT know this.

>> We are lucky that they decided to vandalize instead of hiding something.

> We do NOT know this.

Well, at least we know that the attackers wanted their activities to be noticed at this point in time.

I have reasons to suspect that this is a compromised account, in which case there should be enough forensics info to figure out what the scope of the damage is.

> We are lucky that they decided to vandalize instead of hiding something.

How on earth do you know they haven't done both?!

Git commit hashes are pretty safe.

To verify old hashes, with a backup. But anyone with access can just push a new commit with an arbitrary author field. Without code signing it'd be very difficult to tell who actually pushed it. (Probably GitHub has logs you could check, but if the attacker could get on the corporate network it'd just be a NAT IP, and more sophisticated attackers can spoof or MITM)

Aren't they changing the hash algorithm specifically because they're no longer safe?

Theoretically no longer safe, but to insert a compromise commit matching another one that looks like normal code is close to impossible.

They still pretty visibly log the push event with the identity of the user whose credentials were used, so what you're describing is pretty easy to spot.

> Always sign your commits!

Unfortunately PGP is fundamentally broken. Any identity (email address) can be trivially DoS'd by anyone, because the keyservers are (by design) write-only databases which anyone can add to.

https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d695...

I think that actually means the notion of keyservers are fundamentally broken, not PGP.

I don't need a key server for me to sign or validate that something was signed by who it said it was if exchanged my keys via a secure means.

GitHub’s PGP signing feature doesn’t make use of keyservers. You have to explicitly add a PGP key to your account for GitHub to acknowledge that your commits are signed.

Are these not just mirrors though or do they actually use git? Woulda thought they used SVN primarily since they pull from Debian? Or do they just pull tarbals... And this all leads to me wondering how they do their process... Anybody got good docs on how Ubuntu or similar distros that base off a parent distro do their work?

Debian stores packaging in Git too, these days, sometimes exclusively.

That’s true, but somewhat misleading. Though most Debian maintainers use a VCS for packaging, it’s incidental to what actually ends up in the Debian package tree. That’s all still based off of tarballs submitted off of a developer’s machine.

Much less misleading than the previous post.

> Woulda thought they used SVN primarily since they pull from Debian

First, I would like to challenge the assumption that Debian primarily uses SVN for packaging. We can get a rough idea from the use of Vcs-$name package metadata in the testing distribution:

  </var/lib/apt/lists/*_debian_dists_testing_main_source_Sources.lz4 unlz4 |
    sed -n '/^Vcs-Browser:/n;s/^Vcs-\([A-Za-z]*\): .*/\1/p' |
    sort |
    uniq -c

On my system this gives:

      4 Arch
     72 Bzr
      5 Cvs
      5 Darcs
  24168 Git
     21 Hg
     21 Mtn
    603 Svn

Which shows that git is the overwhelming favorite and about 40 times more popular than SVN for packages which provide VCS metadata.

The sources can be pulled directly from Debian as tarballs using [syncpackage]. Ubuntu maintainers are free to use their preferred VCS for maintaining Ubuntu-specific packaging. Using the same VCS as the Debian maintainer (or upstream developer) is often convenient, but not required.

[syncpackage]: https://manpages.ubuntu.com/manpages/precise/man1/syncpackag...

Surprised to see the same number for Monotone than for Mercurial.

It looks like they have already recovered everything.

Two people have been removed since this occured.

does that imply Ubuntu code might've been compromised , and some trackers / keylogger codes planted ?

Uh, this is worrying.

Reply on news.ycombinator.com