Kernel.org has been hacked (see site news) | Hacker News Reader

Kernel.org has been hacked (see site news)

kernel.org

167 pointsKonradKlause56 comments

From a pure git perspective, the attacker(s) will have a very hard time rewriting any commit history, as changing a commit's SHA1 hash will trigger a cascading effect on the hashes of all child commits on down the DAG from the one that they change.

Anyone with an existing clone of the repos would immediately know the repo has been corrupted when doing a pull (although fresh clones wouldn't be as lucky of course).

If the security at kernel.org has been breached, then I don't know what to think about me, the little guy - who runs a small VPS - or about the small start-ups out there. I guess having secure systems is not that easy after all, and people overlook this or fail to allocate enough resources to this process.

Any system is penetrable with enough motivation or resources. It's just a cost of doing business. Be happy you're under the radar, Sony wasn't so lucky.

The bigger you are, the more servers you have to manage, the more services might provide a way in, the more users have access and might have their credentials stolen... it's actually much easier to be secure when you're small.

The real money-quote from the original article was:

We are currently working with the 448 users of kernel.org to change their credentials and change their SSH keys.

With 448 users, something like this was inevitable...

I wouldn't call a root exploit inevitable just from user count.

FTA: "We believe they may have gained this access via a compromised user credential".

I hope this is true, and the access was not gained via some unknown vulnerability.

It sounds like they got in via a regular account and somehow escalated themselves to root: "how they managed to exploit that to root access is currently unknown and is being investigated."

Simple social engineering of mundane user accounts is not worrisome. Escalating any old user into root is.

After reading the LWN security page for years, I don't consider root escalation as a difficult problem. There seems to be a never ending stream of vulnerabilities. That layer of security is a useful goal but I assume user level access also provides root level access.

Well, the effort that goes into breaking into a system is hopefully proportional to the payoff. This more than anything else is why there is quasi-infinitely more Windows malware than for any other system, or why the idea of a national key registry (back in the Clinton days) was such a horrible idea, the payoff in compromising it would be too high. So hopefully you can scale the resources you apply to security with your profile and such.

I have a question. They note they have notified officials. I've tried notifying the FBI after our sites have been hacked, but they are not interested in it. Is there some other place to report? Does anyone really investigate this stuff?

From a global security standpoint, kernel.org is an important site. Hacking the authoritative source for the linux kernel has serious implications for anyone who runs linux, which is lots of people probably including law enforcement.

Note that, as mentioned on the site, actually doing this is much harder than simply rooting the servers.

It's useful to make a report as soon as you notice something wrong, even if they don't do anything right away. If it turns out to be more serious than you (or they) first thought, it's nice to have a timestamp on a detailed report from when things first looked funny.

Does anyone really investigate this stuff?

Not unless there's a ton of money involved or some important asset is at risk. This type of thing happens much more frequently than they can manage so they prioritize by severity.

I don't know what it's like in the US, but in my home country you're supposed to report it to the police -- just like you report the burglary or mugging.

id like to point out that while you can't modify tagged+gpg signed tag commits, you can modify non signed commits (aka any commit that isnt a tag) and it'll get signed when it gets tagged.

that's also assuming that tags are gpg signed, which they often are not,even on kernel.org

non signed can be actually tampered with and they do need to check the code thoroughly with 3rd party older archives which is a PITA.

Code signing > x, including per commit signing mr Linus T and GIT maintainers.

Note that they were against per commit signing because "when you sign the tag, you sign everything so its ok".

Except you wont read all the patches you sign when you sign the tag, and if any has been modified, as explained, you don't know. Again, per commit signing solves it.

Food for though I guess.

It's not a question about whether you read them; code review isn't the mechanism. The modified commits will have different hashes and thus cause merge failures for anyone that already has them in a local repo. At the very least, that's going to include Linus and everyone downstream as far as the original author. None of these people need to do anything to notice the tampering beyond trying to merge from mainline.

That may not be perfect, but it sounds pretty robust to me.

it does not matter, you can add a new commit (or modify an old one anyway since it'll work fine - but lets say you notice it, so we add a commit) you make the commit look legit and all, its going to be pretty difficult to say where its coming from and which has been tampered with, since, of course, linus himself will have it when he pulls it.

and that's not just linux.git, but also the ton of other .git's in there, which linus will pull from and will not read every line of.

The caveat here is as real as it can get. usually when it looks "slightly complex" people overlook the security aspect "no one is going to do such an attack"

then the attack happens, often unnoticed, because the attackers aren't lazy if there's a hole they use it.

Somebody might not notice a malicious commit when they pull the tree, but the maintainer will definitely notice the next time he pushes to that tree. Having a push rejected because it's not a fast-forward gets your attention every time.

But what value does adding an arbitrary unconnected commit have to an attacker? The only sane goal would be sneaking exploit code into a "released" version that will be shipped.

In order to do that, you need to somehow get Linus to merge it successfully. And that doesn't work per git's design.

If all you want is to have "exploitable" commit IDs in the kernel git, you don't need to do anything sneaky at all! They're already there, being historical bugs that have since been fixed.

They gained root access. So some one could do

sudo su - linus

cd /path/to/linux-2.6.git

modify/exploit code and git commit

I for example do a daily git pull - I will get the the tainted code and not notice it at all.

NO?

But that's just HEAD. No offense, but so what? Clearly anyone (sane) running off of a development branch isn't doing so on a security sensitive system. It's an exploit, but only of development systems and a tiny handful of hacker's personal boxes. The value there to an attacker is very low.

What you can't do is inject code that ends up in 3.1-rc5, because before that gets tagged Linus (and a lot of other people) would have to do a merge that would fail inexplicably.

But to be clear: yes, with access to the repository you can (for a brief moment until it's noticed) inject exploit code that will be picked up by anyone building and running an untagged branch HEAD.

Ubuntu does daily kernel builds and many people run that kernel - to see if some bug has been fixed, to get support for their newer hardware etc. Lots of reasons perfectly sane people can end up testing a daily build.

The code would not be signed by Linus; you do not know his private key.

If I as a developer then pull and verify signatures, I would note that commit was unsigned and expect the commit to be compromised.

But you cannot sign commits in git, only tags, right?

Linus and co. do read every patch. Either Linus or somebody Linus trusts absolutely, has read each patch going into the kernel. Also, release tags are signed, for every single kernel release.

The way git works is that every new commit depends on every commit before it. Since no code is brought into the trees without checking the commit IDs, there's no way that any non-tagged commit could be modified without somebody noticing.

I understand that you don't like git, but you might want to read up on how the kernel is developed before making spurious claims.

Thank god for Git.

Thank Linus for Git.

But you repeat rhygar.

"My name is Linus Torvalds and I am your god." - Linus Torvalds http://en.wikiquote.org/wiki/Linus_Torvalds

I don't do any development and I've still heard about git's reputation.

edit: Why is this being downvoted?

because it doesn't make me smarter to have read it.

I pointed out that git's good reputation is known even outside developer circles. It might not be the greatest contribution ever, but is it really worth -x?

Anyone know how this affects the ArchLinux packages? kernel.org is still the fastest mirror around for them.

Probably not at all (the Arch userbase isn't much of a target). That being said, pacman doesn't implement package-signing last time I checked so it certainly would be easier to pass off bogus packages than with other distros.

Pacman now supports package signature verification (it'svery recent news.)

I wonder what was the reason behind this?

>However, it's also useful to note that the potential damage of cracking kernel.org is far less than typical software repositories

What about code that's hosted on kernel.org itself? Isn't kernel.org a source for the public to get the kernel and not git?

http://www.kernel.org/pub/ ftp://ftp.kernel.org/pub/ rsync://rsync.kernel.org/pub/

It would be easy for the exploiter to insert trojaned/rootkitted kernels into those places.

Currently all files get verified. The kernel.org admins have digital signatures of all files...

Hopefully those signatures aren't on the compromised machines?

Of course not.

As one of the users@kernel.org members I can tell you that the kernel.org admins are very competent.

if md5 of files exist then there seems no prob.

This is a valid argument against open source operative systems running top clearance environments such as the military / police / government agencies. If this modifications had gone undetected for a few months, it's possible that the compromised code could have made it into a lot of critical systems.I am a Linux user, but I remember this being a Microsoft argument in the past for promoting their OS to run in government agencies.

no modifications to the code appear to have been made. Like they said, that would be hard/impossible because it's all signed off by Linus in git, so ever if they compromised the server it gets them nothing. They'd then have to compromise some accounts and submit patches and still get them approved.

This argument is completely bogus. I could just as easily have happened to any one else including Microsoft, and in those cases we might not even have heard about it.

It already has happened repeatedly to some hardware vendors where an actual payload was injected into their drivers, and they weren't open source.

Between open source and git it's dramatically more likely an injected payload would be detected long before dissemination could take place.

Go with "impossible." Gov't agencies don't just upgrade every time a new kernel comes out.

Did you actually read the page? Because it says exactly the opposite. It says that it would be impossible to modify the source without hundreds of people noticing immediately.

No, it isn't. Imagine someone got a developer's username and password at Microsoft. Then they logged in and managed to escalate themselves to Administrative privileges on the box that manages Windows source control via a security vulnerability. They then injected some backdoor access into Window's networking code for an upcoming patch. I would argue the chances anyone at MS would have noticed this is actually lower than compared to a distributed environment (git) that is designed from the ground up to catch these kinds of things. There is nothing about a proprietary system that makes this kind of thing any less likely.

How? Don't commercial OS vendors get hacked?

Proprietary codebases aren't fortresses.

edit: And what ars said.

edit: And the chorus.

How long might it take for a problem to be detected were compromised Windows code to make it into critical systems?

> we currently believe that the source code repositories were unaffected

So for now it looks like no compromised code has been distributed.

kernel.org hosts more than the GIT repo.

http://www.google.com/search?client=opera&rls=en&q=d...

How would a proprietary operating system be better?

We'd not know about it :)

What is the latest update?

> Intruders gained root access on the server Hera. We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.

This is more related to weak security policies than OS security flaw.

It's more related to having multiple users with system access. Your security policy now extends to the security policies of all users and user systems. Password strength, system integrity (keyloggers, etc.), token security, and the like.

Hard security is hard.

Reply on news.ycombinator.com