Github SSL replaced by self-signed certificate in China | Hacker News Reader

Github SSL replaced by self-signed certificate in China

189 pointsteawithcarl75 comments

CHINA NEWS -

Man-in-the-middle attack on Github. -- http://bit.ly/Vqh8zJ

Fake GITHUB cert -- http://www.mediafire.com/?zx6eno648axz7bh

Twitter (Github attack news) -- https://www.twitter.com/search/?q=Github%20SSL

Firstly, thanks to GitHub using HSTS on github.com (although not www.github.com), the certificate error will be fatal in Chrome and (I believe, but haven't checked) Firefox as long as you have visited GitHub previously.

(It's not preloaded HSTS so it would have to be learnt from a previous, unattacked connection.)

I know that the unbypassable errors for some sites upset the more technically minded people, but I think that incidents like this show its value.

The CloudShark trace shows what appears to be Firefox connecting to the GitHub IP address, but the server clearly isn't GitHub from the config. The server appears to be configured to accept the client's ciphersuite preference, but doesn't support DHE nor ECDHE.

The server is also only 9ms from the client - that's clearly not crossing any oceans. I'd also guess that the server is overloaded at the time because the ServerHello (which doesn't take significant processing to generate in this case) takes 900ms to come back.

Sadly, it appears to show the user overriding the certificate error and talking to the server anyway :( Hopefully that was a fresh FF install just to see what would happen (which would explain why HSTS didn't prevent the override).

Lastly, the certificate appears to be self-signed, but the Authority Key Id doesn't match. One assumes, based on "OpenSSL Generated Certificate" that OpenSSL was used, but the person may have had some trouble. I'd guess that they generated a CA certificate first (with the same Subject) and then signed the certificate in question as a leaf. Many of the tutorials that you'll find online are for that sort of setup so perhaps they weren't very familiar with X.509 certificates.

didn't know of HSTS, thanks.

where would one keep up with stuff like that other than keeping up with new rfc's?

agl's blog is a good source: http://www.imperialviolet.org/

OWASP has lots of useful security info, https://www.owasp.org

I maintain a complete guide to SSL/TLS deployment:

SSL/TLS Deployment Best Practices https://www.ssllabs.com/projects/best-practices/

You may want to integrate some of the advice from here in your HSTS section:

http://coderrr.wordpress.com/2010/12/27/canonical-redirect-p...

It seems even github is susceptible to this. That is, for people who type www.github.com into their browser rather than github.com. They both did the redirect wrong, as well as left off HSTS of https://www.github.com.

To clarify, this looks more like someone turning off SSL access to GitHub than a proper MITM attack in the traditional sense.

The certificate in that link is just a self-signed certificate, not something signed by a CA:

Issuer: C=US, ST=Some-State, O=github.com, OU=github.com, CN=github.com Subject: C=US, ST=Some-State, O=github.com, OU=github.com, CN=github.com

So your browser will warn you that you are not making a secure connection. Firefox users, for instance, will have to make 5 clicks to get through that warning and visit the page.

I think "China turns off SSL access to GitHub" might be a more appropriate title.

Thanks, Moxie - The original article in China (by top-notch geeks) wrote 中间人攻击 which translates directly to "man-in-the-middle attack".

Appreciate your insight - thanks for weighing in on this.

I don't think there's proof of China doing anything here. Perhaps someone on OP's coffee shop WiFi is poisoning arp tables and forging certs.

This seems really sloppy for China. Without further proof, I don't think it was the govt.

Agreed there's nothing in the data to directly suggest government involvement.

It's only "sloppy," though, when conceptualized as a MITM. China does have an extensive history of censoring access to sites, and recently censored access to GitHub entirely IIRC. It could be that they decided to block SSL access, but allow HTTP access, and this is how they implemented that.

It appears that my contrived scenario is incorrect. The incident seems to have occurred at several locations across the country. Assuming this is correct, it does appear to be a government action.

https://twitter.com/0xabad1dea/status/295606501798719488

Everything in my bones (25 years, 中文研究, China research) tells me the China government is directly involved with this. China is corrupt beyond belief, and any smaller destabilization can lead to further problems.

I agree that this may be a further extending of the "New Years train ticket" block on Github.

It may also be new toying after the recent "experiment". Leaving Github without SSL inside China still makes trouble - China's insidious corruption at the very top is subtle, incremental small steps, all designed for the "long game".

It may also be raw mercantilism ... as with Google, Twitter, and Facebook long before this.

As an sort of old china hand, china is corrupt but not beyond belief, there are plenty of countries that are much more corrupt, even India is worse than china and they even have democracy.

The level of sophistication that the GFW seems to be achieving is disturbing. We've had certificate attacks before, perhaps they are testing something out that will be deployed more broadly to solve there "gmail" problem?

How does your 25y China research made the Chinese character wrong?

It should be 中国研究 but you said 中文研究, which means Chinese text research.

SSL infrastructure was already known to be best treated as globally compromised.

If this is real, then the Chinese may have collected some interesting statistics regarding the percentage of developers who don't care about certificate issues.

I didn't see the cert appear over the wire myself, but I'm in China at the moment and spent part of this week in communication with the Gentoo and Debian release engineering teams suggesting they revisit perceived issues in their respective key distribution processes and documentation (issues focused on automated validation of install media; not individual packages). Gentoo was already working on it and Debian got back to me pretty quick. I don't feel I was wasting my time now.

Really, we need a key distribution and trust anchor solution for the masses, as Moxie has spoken about, that includes 'trust agility'. IIRC the latest iteration of his proposed solution there is http://tack.io/

It's worth pointing out that many governments have MITM and warrantless surveillance systems, not only the Chinese. For more background see http://wikileaks.org/spyfiles/ which summarizes "Mass interception of entire populations is not only a reality, it is a secret new industry spanning 25 countries." and Jacob Applebaum's keynote at 29C3, https://www.youtube.com/watch?v=QNsePZj_Yks (Youtube is also banned in China).

(edit: posted beneath this thread as moxie is one of the community's most respected parties in this area and just posted, but had some extra info in here! :)

And this only lasted for a few hours. Users connecting to github saw perfect tlsv1 session without github sending its real cert. This means github was routed to a reverse proxy for the mitm test.

This reminds me the Firefox certificate "bug"[1] two years ago. A China certificate root server was added into trusted servers in Firefox and Chinese hackers started to submit bug report regarding this, since people don't trust certificate servers run by China government. Man-in-the-middle attack was exact what Chinese hackers worried about.

If they put this fake certificate in a certificate root server that's in the trusted server list, they can easily get anyone's account who's using affected browsers.

It's weird that they start with Github. It's not a website that's popular among human activists or any other people that China government might be interested in. Instead, it's popular among programmers and hackers, who are the main group and forces in China to help people bypass GFW to access blocked content. I suppose this attack might be what the government uses to test reaction and capability of hackers.

Seriously, this is really, really, bad.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=542689

EDIT: added link for bug report

If you actually look at the certificate, it is self-signed, not signed by a malicious root server. Your anecdote has nothing to do with this current attack.

You are right. This one has nothing to do with root servers. But they are already able to fake certificate national wide. It's only one step from adding it into root server.

Maybe having access to the sourcecode of a site or app = access to the site or app.

Spoofing GitHub's SSL certificate is a step in the direction of inserting espionage-style backdoors, as GitHub permits HTTPS read-only checkouts of repositories.

I'm not suggesting that this will be free of problems, given how particular Git is about checksums, nor am I certain what methods they would use to acquire SSH commit access for altering repository contents to affect the rest of the world.

Still, it's absolutely a necessary prerequisite to altering data from GitHub - and, if they acquire SSH keys, altering data at GitHub. Both possibilities are terrifying.

GitHub now [allows HTTPS pushes][1], so a man-in-the-middle attack on a single connection would be sufficient to push backdoors. In any case, can't the intermediary just add another SSH key to the account?

[1]: https://help.github.com/articles/pushing-to-a-remote

Some possible reasons:

GitHub is popular among developers.

Many developers have faster computers and more computers per household than others that aren't in IT/development.

Many developers, though they pretend to be security minded and focused at times, often have a more lax attitude in practice than system administrators, especially at home or working on personal projects.

Some developers have more usernames/passwords and certs to other servers available than non-developers would.

Some developers have access to other confidential data or newer technology that could be of practical use.

My sense is that there is a correlation between geeks and activists, so I think it's a "logical" group to target for further "attention"/persecution. It also makes sense to want to inject flaws in project code. (Even though git has hashes, the hash can be fine if the injection happens on commit.)

I have always imagined that github would be a huge target for the chinese... for industrial espionage. they could get the source for private repositories for many startups and successful companies.

Don't forget the ability to inject code into repos on behalf of someone else.

If you've been logged into github and you told your browser to ignore the warnings then better keep a really good eye on your commit log, lest something pops up that you didn't actually put in there.

And change your password at the earliest opportunity.

I doubt very much of value is on github.com as opposed to more secure Github Enterprise installations behind company firewalls.

Uncounted open source projects that end up being run with root privileges are on github. It all depends on how you determine value.

If value is defined as the number of machines that can be backdoored, communications intercepted and so on then it may very well be that that other form of value will be realized in good time as well.

Tools for bypassing the firewall are often distributed on github, so it makes sense as part of lockdown.

If CNNIC is complicit in a MITM attack there will be a paper trail (namely a certificate signed by them) proving their involvement. To this day nobody has produced a cert signed by CNNIC that was used for a MITM attack. There's no reason to believe that CNNIC is bad/evil/whatever other than their affiliation with the PRC.

Disclosure: I work for Mozilla, but not on security.

CNNIC is notorious for producing malware and many other things. Some is list here http://en.wikipedia.org/wiki/CNNIC

Having been in the internet industry in China for more than 10 years before I decided to move to the US I would say CNNIC is just as evil as any government organization in China if not worse. There are some well known rules regarding CNNIC: don't buy a .cn domain, don't install any software from them and don't trust what they say.

BTW non government organizations are technically banned in China. You need to apply for a permit and you seldom get one.

Thanks for the reply. Glad to see some guy from Mozilla here!

You are right there's no reason to believe that CNNIC has to be bad/evil/whatever, but we cannot assume CNNIC would never be bad/evil either. We have to look at its history, and what's behind it.

When something turns bad, it won't throw out an announcement beforehand. And we need something that can handle this in time.

I'm thinking about something like this: in addition to the trusted root server list, keep another list, e.g., less trusted root servers; besides normal update program, put a special piece of code in the browser that allows firefox team to delete certain root server from less trusted root servers on the air. By "on the air", I mean not needing a full version update, or restarting the browser. It can warn the user, but it should not be possible to canceled the deletion.

That's something that we've considered doing. I'm not sure what the current state is. I suspect there's a bug somewhere on bugzilla.mozilla.org for implementing a cert blocklist that does not require a full update.

> It's weird that they start with Github.

I don't think it's safe to assume that they started with Github.

FWIW, I always try to remove the CNNIC root cert from my setups. It's not that hard to do (except for iOS devices) and I have never had an issue since I don't frequent sites that would be signed by CNNIC.

Fortunately, it looks like Chrome prevents users from clicking past the ssl errors. This Chinese chrome screenshot[1] shows that there's only a back button (for English versions of this page, see [2]). Unfortunately, it appears that IE allows users to click past the errors[3]. I'm also interested in how the 360 browser[4], which has been gaining market share in China, handles the error.

Does anyone have any theories why China would use a self signed certificate when it's very indiscreet? A lot of users will just click through if given the opportunity (around 60% in chrome before new security measures prevented it[2]), but I doubt many users with truly sensitive private repos would do this.

[1] https://twitter.com/GreatFireChina/status/295236912594186240...

[2] http://www.imperialviolet.org/2012/07/19/hope9talk.html

[3] https://twitter.com/chenshaoju/status/295139636718743552/pho...

[4] http://news.ycombinator.com/item?id=4499168

What bugs me about stuff like this is that there will always be mercenaries, guys just like you and me that will do anything as long as it pays. The Chinese government wouldn't stand a chance if they had to do this stuff themselves. Mercenary coders are nothing new, we have them in every country (and sysadmins, companies and so on).

But you have to wonder what goes on in their heads, what mindset would prompt you to sell out like that.

I kind of get what you're saying, but I think there's a lot of hubris in thinking "my profession is special, only good guys (can learn to) do it". Craftsmen have been complaining about the wrong people getting into their trade for as long as there have been crafts.

We should be worried about mercenaries when it comes to malware, Russian botnets, and the like.

In the case of the Chinese government, I believe the bigger problem is patriots. They honestly believe that they are doing the Right Thing. So did the people who wrote Stuxnet.

Just to be clear, Stuxnet is claimed to have been written by the US government/Israeli Government, not by the Chinese.

Yes.

And my claim is that it was likely written by people who consider themselves patriots, not mercenaries.

Some speculation as to why they're targeting Github:

- the Chinese gov't is trying to identify users/developers of train-ticket-purchasing bots [1]

- they are is interested in capturing some intellectual property contained in private repos

- it's just an exercise to watch & learn how computer-literate users circumvent a MITM attack

Given the recent Github blockade, I'd go with the first.

[1] http://www.techinasia.com/github-blocked-china/

Many are mentioning the Chinese government, but are there any signs that they are involved?

How do we know that this isn't just a hijack of a Chinese isps dns server or something similar? Maybe the same that happened to Goolge Morocco just a couple of days ago ( http://arabcrunch.com/2013/01/breaking-google-morocco-google... ).

FYI there is a petition on whitehouse.gov to deny those people who work on GFW entry US. https://petitions.whitehouse.gov/petition/people-who-help-in...

We should deny all those companies that have developed tools of censorship access to government contracts and deny them the right to work in the United States ... oops, there goes almost every single major networking manufacturer!

Cisco: http://newsandinsight.thomsonreuters.com/California/News/201...

Juniper also implicated: http://www.cfr.org/china/us-internet-providers-great-firewal...

Blue coat: http://online.wsj.com/article/SB1000142405297020368750457700...

Websense: http://opennet.net/west-censoring-east-the-use-western-techn...

The list goes on ...

It sucks my client will be able to say he was right about now allowing source code to be hosted on github.

(We ended up setting up a gitlab box and it works just as well)

I find this a sad commentary about your (and certainly others) relationship with your clients.

This is utter rubbish, hosting your own git server(s) is easily justifiable.

The client declining to hosting code on github should be enough justification.

There is nothing wrong in insisting the repository to be hosted on the client metal. The way things were done in the ancient times before the cloud to become the favorite buzzword of the month.

I agree. The cloud isn't new. VPN's have existed for ever as well as client-mainframe.

The perception that the cloud is somehow re-invented is in fact the saddening thing. It's just more accessible and faster than in the fast, but ultimately you can't manage infrastructure by abdication and farming it out.

There's always an uneasy balance between security and convenience.

My client would never allow their source code to be hosted in the cloud either. There's really no point fighting them on this, since they employ security and legal teams for just this purpose. The external marketing partner is not going to trump that. Ever. And there's nothing wrong with this.

I'm not sure what's sad.

There's always a precarious balance between security and convenience.

The customers own their code. They have been around longer for more oscillations between the "cloud" and bare metal than most people who have an opinion on it.

They're forward thinking enough to not embrace the cloud, or bare metal, and instead are exploring hybrid cloud technology that can run on a combination, and more importantly, move dev ops between different environments.

One reality is many people who blindly think it's ok to put everything on in the cloud, don't always know of the reality of liability, or have a relationship with any codebase and the associated IP for more than a few years.

What exactly is he right about? China hasn't done anything that you can't do on a local WiFi connection. They grab the connection and put a self-signed cert to it. Security is still intact. The only way this hurts anyone is if they blindly trust all certificates, in which case they're screwed anyways.

I'm not saying he's right about anything -- he'd remind me of his preference to host inhouse.

Personally, I use a mix

In China now. Ping to github is 280ms. The cert I receive is valid. So either they stepped up the game or it isn't universal.

Even the Chinese gov does A/B testing.

It's more likely capacity limit than A/B testing. Returning arbitrary response is consuming more resource compared to cutting off connection (sending TCP RST).

In a Hotel in Beijing. Ping around the same, Chrome does SSL and shows no error.

Same thing in Shenzhen (China Telecom). Everything seems to be normal.

I guess it was a planned public test or technical verification of certain larger project about live traffic decryption on national level. There really isn't a plausible political reason of github being targeted. Maybe targeting github can get data about user response or verify scalability of the infrastructure given recent high https traffic volume from China to github.

It seems strange that they would target a site whose users are exactly the people most likely to detect and understand (and make a fuss about) internet peculiarities.

I was wondering when Github was going to start supporting HSTS and 2-Factor Auth. I'm betting that it gets bumped in priority after this event. Nothing like an incident to move along security requirements!

http://dev.chromium.org/sts

https://www.duosecurity.com/features

Why use duosecurity over something like RFC 6238 and RFC 4226?

I don't think either would help in this situation. HSTS helps prevent SSL stripping attacks, which whoever is denying SSL access to GitHub doesn't need to bother with.

2FA serves as an annoyance to phishers, but whoever is doing this network attack has direct access to your session cookie.

We've supported HSTS for for well over a year now.

I should report that from my Shanghai based VPS I don't get this problem.

Perhaps this is localised, or was just a trial for a few hours.

https://gist.github.com/4650029

Read the Twitter stream - affects people inside China.

Still, shows the depth and corruption the China (gov't).

https://twitter.com/search/?q=Github%20SSL

How would this fake cert work? It doesn't seem to originate with a root or have any associated information. Do browsers actually accept certs that look like this?

It would work if users accepted the self signed certificate upon receiving the warning.

http://pic.twitter.com/xOxhoyzp

Censorship in China is a big business, for the companies that have connections to the officials, for the universities that rely on the funds, and for the officials who use this as a stepstone for their career. It is so common that you can even find many fresh graduates who worked for the GFW in the job market.

How long ago did this start? I wonder if it has anything to do with some of the issues with Github I (and others) were noticing on Friday[0]

[0] https://twitter.com/davidbalbert/status/294941563673522176

Here's another (safe) "tweet" by China geek alerting the situation.

RT @chenshaoju 无锡电信实际测试GitHub已经遭到SSL中间人攻击。 http://bit.ly/X49oPK

How did GitHub learned that there were man-in-middle attack?

The natural result of the "code is speech" argument?

Bad news, access to github is unsafe for people in china now.

Reply on news.ycombinator.com