Main GNU source repository server compromised | Hacker News Reader

Main GNU source repository server compromised

savannah.gnu.org

135 pointsthere39 comments

I see a lot of angry comments on their webpage, people asking why somebody would do this. Well, the real answer is probably - simply because he/she could. And to all of those wondering how anybody could do this to a project that hosts free software, well, the attackers don't really care. They saw the exploit and used it. Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection.

Maybe someone had an idea for putting a vulnerability into a GNU utility.

Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection.

Seconded. But then again, the GNU source code is mostly C, isn't it? They're used to doing things that most developers would consider intractable and therefore impossible to do safely: such as comprehensively sanitizing inputs so you can safely construct a query string out of them.

"Hey, those web guys say sanitizing inputs is effectively impossible. Should we really be doing this?"

"Snort. That's what they say about manually deallocating memory, too."

"Oh yeah! What a bunch of wimps! I'll get right to work on the query builder."

Going further: how about a vulnerability in every program compiled with gcc? I'll just leave this here:

http://cm.bell-labs.com/who/ken/trust.html

I'm pretty sure savannah is php, forked from the original *forge code, so not really a fault of the people who wrote ls and gcc.

People in 1997 never expected that <insert project here> could be vulnerable to an attack as simple as a buffer overflow; after all, all it takes to not be vulnerable to overflows is "counting", and who has problems counting with a computer? Tens of billions of dollars in losses later: do you feel safe hitting arbitrary pages with your web browser?

The code might be much worse than anything they would have written themselves. http://en.wikipedia.org/wiki/GNU_Savannah explains they started with a fork of the SourceForge code, and they seem to still be on it (the Python/Django rewrite isn't complete). It's not obvious what SourceForge was written in, but http://sourceforge.net/apps/trac/sourceforge/wiki/Open%20Sou... shows an awful lot of PHP dependencies, and the site dates back to 1999, a time when the worst of PHP's myriad design flaws (such as register_globals and magic_quotes) were widely embraced.

From the comments on the linked page:

Date: Tue Nov 30 01:34:15 2010 Pseudo: mjflick - Savannah Hacker Comment: re: HappyCrow,

Only one project was targeted in this attack.

Second, a postmortem will be forthcoming, as well as more information shortly from the FSF staff, who are planning on making an official announcement about this.

Ouch. This is bad.

It illustrates how much of a pain SQL injections are, if they can affect the GNU project which has some of the most incredibly talented hackers worlwide...

Props to them for having a working backup strategy though.

Actually, no... you have to be pretty ignorant to be subject to them. Use bind variables in your queries and you are done. That it's still one of the top 2-3 exploits tells you how horrible most developers really are.

Not all talented developers are talented or even good at security.

No. You are not done with SQL injection just because you use "bind variables". Plenty of things can't be parameterized that are nonetheless subject to user influence. Click the top of any reverse-sortable column in a web user interface to see one obvious example.

Parameterized queries are a good thing, and you should use them, but I feel like I've had to be a broken record about this over the last week: they are not magic anti- SQL injection fairy dust. Our whole team spends most of its time looking at smart people's code, and we routinely find SQL injection.

Plenty of people who aren't ignorant of SQL injection manage to let SQL injection slip through. Knowing how SQL injection works and being able to devise and implement engineering procedures to reliably prevent them are two very different things.

Click the top of any reverse-sortable column in a web user interface to see one obvious example.

People don't do those in JavaScript?

That's what I did a decade ago. Well, then I moved it back to the server after we got tired of the performance problems that JavaScript had back then. But today it wouldn't be an issue. And when we moved it back, we were careful not to have an SQL injection attack. If memory serves we actually did the resort in Perl. (In our defense, much of the data we were serving lived in flat files, or were generated on the fly from a compute server, instead of coming from a database.) However at another company I had the same problem, and I did the obvious "process CGI parameter, insert appropriate ORDER BY statement". Where the definition of appropriate was by column position, from which I worked out the field to sort by, so I didn't have to trust the client for the name of that column.

I should back up. We didn't have any SQL injection attacks that I knew of and were reasonably careful. But that code base did not get audited, so I can't really know that. However after the next company that I worked for got bought by eBay, they did a penetration test on us. The worst thing that they found was an open redirect that could be used to let a spammer construct a link to any web page with us as the referer.

I'm happy to use this as a testament that programmers really can avoid SQL injections. However their surprise that they didn't find any SQL injections in our code supports your claim that most teams fail to do so successfully.

I'm not sure what you think I'm arguing; that it's impossible to avoid SQLI? Of course not. All I'm saying is that parameterized queries aren't the end of the story; the teams that believe they don't have SQLI because they use parameterized queries are the ones that are going to lose their apps to SQLI.

What is the full story?

It seems to me that two tricks nail it. First use parametrized queries. And secondly if you have information you need to send/receive from the client that isn't easily parametrized, have a limited list of possible things that can be accepted back, which is checked in code while building the query.

If you're doing those two things, I don't see how much work it is to avoid SQLI. Furthermore if you're using a reasonable ORM, then you should get both of those pretty much for free. (Well you have the overhead of learning the ORM itself.)

There's almost nothing wrong with your second "trick" (when we write recommendations, we usually suggest people never allow users to directly write syntax, and that they structure their app so that the parameters the user sees couldn't possibly land in SQL; use "1" for "ASC" and "0" for "DESC", etc).

But it's not nearly as powerful a statement to say "use parameterized queries and then do everything else right" as it is to say "just use parameterized queries and you won't have this problem", is it?

How about "only pass user-sourced data in parameters"? So anything that isn't a parameter is coming from the app, possibly as a response to user data, but not actually from user data. This statement seems just as powerful as the original. Anything misleading about it? Somewhere you can't use parameters and can't simply use app-supplied query pieces?

You see how we're converging on "just use proper input validation", which is the all-time least useful piece of advice in software security?

Because input validation is hard to do. A rule to not pass input EVER (outside of parameters), even in a supposedly validated form, is something different.

People don't do those in JavaScript?

when you have to add pagination to sorting, you have to send the sort column to the database server.

lol, pwned twice in 7 years, "incredibly talented hackers" lololololo

I believe it's more that they are a visible target, being in the media and spreading their message as much as they do. Were they a small company that hosted internal webapps (for example) they probably wouldn't have been targeted as much. Imagine, for a moment, if these two attacks in seven years were successful, how many unsuccessful attacks there were as well.

Most, if not all, software after a certain point of complexity has vulnerabilities (yes, even mine). This has absolutely nothing to do with how talented their hackers may be and more with the inevitability of someone, somewhere, working hard to find, and abuse, exploits.

edit: I typed more than I intended to here, but my gist is that you're really taking light of the situation here, trivializing all of the work that has gone into gnu projects when you quote "incredibly talented hackers" and follow it up with a useless throwaway phrase like "lololololo". I thought HN was better than this.

SQL injection

In any sufficiently large group of people, you're going to have assholes. To dismiss the people who've put so much good work into the GNU project would be offensive if it weren't so foolish; it says more about the poster than about the GNU folks. Also, consider the source:

http://news.ycombinator.com/user?id=konad

Two previously banned accounts. It looks like HN is better than this, as a whole.

All the programmers are working on other web-based stuff. We need more volunteers, how about YOU konad?

Ok... I hope I misunderstood this, but does anyone see a problem in the order they're doing this in?

    [X] Reset passwords
    [/] Fix SQL injection and look for potential others
    [ ] Implement crypt-md5 support (like /etc/shadow, strong and LDAP-compatible) hashes
    [ ] Implement password strength enforcement

So they're basically forcing password resets, even though people can use "password123" in plaintext on a site that's not completely audited for vulnerabilities?

You eliminated step one, which clearly states:

[X] Put services online using backup, except for password-based ones (e.g. the web interface)

The other step you omitted is the one where they allow people to log in after fixing the vulnerability.

As for the audit, there's no such thing as a complete security audit, and there's no reason to keep the site down in fear of hypothetical holes.

Sorry, my mistake. I got the website<->other services restoration time backwards. Thanks for correcting :)

Re. audit - of course you cannot be sure, but once you get taken over and your system is offline anyways, it might be a good idea to at least grep the sources for queries and quickly check for obvious stuff. Which is what I assume they're doing.

Question - if they had been using a DVCS which checksums every commit, would they have been safer ? Presumably, you could build a tool that stores a list of checksums and if you had to do a restore, that tool could validate the backup.

Or is it that, with a little effort, DVCS' can be hacked/corrupted fundamentally ?

I don't think DVCS would have helped in any way. It's not the repository server itself that was compromised, AFAIK, but "only" the web interface. This allows password recovery and thus allows people to e.g. make malicious commits from a trusted account, but that would be no less true if there were a DVCS under it all.

agreed - but my question still stands: Take SVN for instance - if I managed to hack into a SVN server, can I change the source code (under SVN management) in a way that would be undetectable ?

Repeat the same question for git, mercurial and CVS.

For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really (as long as you assume you change only the data - clever attackers could serve one repository to the developers and another to .mil/the rest of the world/...) I don't know about mercurial, but I'd wager it's the same as git in this respect.

In either case, an attacker could add arbitrary commits, and the system administrator would roll back to the last known-good backup as soon as the compromise was detected. I don't think DVCSes help much here.

For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really

I don't think DVCSes help much here.

Those statements sound contradictory. Let me try to understand - the distributed aspects of DVCSes would not have helped here. But the incidental fact of checksums in DVCSes (maybe necessitated by the nature of "distributed") does help. Right ?

Not all that much. If I hack your SVN repository, I put something like "if ($password == "1337h4xx0r") $admin = 1" into an old revision and hope you don't notice it when inspecting code. If I hack your git repository, I make a commit with text "spelling fix: invole -> involve" and slip in something like the above. Yes, it's a bit easier to detect, but not all that much - most such breaches seem to be found by system administrators before the developers notice them. (The kernel.org source code breach was detected because some mirror failed to line up with the main repository, for instance)

Of course, a smart hacker would do something less obvious. See the obfuscated C contest for inspiration.

And again, this assumes that I do not have any other access. In reality, I could change your $PATH to invoke a trojaned git binary, or somesuch.

Headline missed a word : "again"

Original issue date: August 13, 2003

http://www.cert.org/advisories/CA-2003-21.html

ftp://ftp.gnu.org/MISSING-FILES.README

                              Moving Forward

All releases after the 2003-08-01 date will have checksums GPG-signed by the GNU maintainer who prepared the release. This assures automatic certification of the integrity of all GNU source from that date onward.

Slightly different, as that compromise was of the FTP servers, which store packaged releases. Savannah is the development system.

I thought more eyes on the source means that OSS is impenetrable. Guess not.

Nobody really cares about helping out with Savannah. On the emacs-devel mailing list every time Savannah is mentioned, it's in the context of "Yeah, we would like to do that but lack the manpower. Any volunteers?" followed by crickets.

What a moran whoever did this.

“But he has nothing on at all,” said a little child at last. “Good heavens! listen to the voice of an innocent child,” said the father, and one whispered to the other what the child had said. “But he has nothing on at all,” cried at last the whole people. That made a deep impression upon the emperor, for it seemed to him that they were right; but he thought to himself, “Now I must bear up to the end.” And the chamberlains walked with still greater dignity, as if they carried the train which did not exist.

Reply on news.ycombinator.com