GitHub hit by DDoS attack
status.github.com
status.github.com
Baidu has Baidu Analytics, a service similar to Google Analytics. In short, a website includes a javascript file from Baidu and Baidu will report some basic analytics to the site manager like how many visitors per day, how much time they spent on average per page etc.
Someone in the middle between a client outside China and Baidu, allegedly it should be the Great Fire Wall, changed the javascript file from Baidu and added some code so that any client executing the javascript file will periodically access https://github.com/greatfire/ and https://github.com/cn-nytimes/. This means any user who is accessing a site using Baidu Analytics will be an attacker to github.
Here is a simple solution: Block any javascript from Baidu if you do not use it. For chrome users, add the pattern [*.]baidu.com. See here[1].
Edit 1: Added a solution.
Edit 2: Format.
Edit 3: Oh, it's not only Baidu Analytics. Baidu Ads' javascript is also being hijacked and changed [2]. Imagine that all sites containing Google Ads use their visitors as attackers to attack github. Now it is literally what is happening to Baidu and its customers (and their customers' visitors.) The javascript is only changed for visitors outside China. This is why people believe that is done by Chinese government --- the only entity who has total access to all out-going routers in China. Since many Chinese users use VPN or other types of proxy to access Internet, they are all considered as visitors outside China.
0. http://drops.wooyun.org/papers/5398
1. http://www.howtogeek.com/tips/how-to-block-javascript-and-ad...
But, really, you can hardly negotiate with Chinese government. I'm pretty sure that they will deny this attack and re-emphasize their so-called Internet policy.
If I were github, instead of a warning message, I would redirect the workload to some Chinese government's website and let them suffer what they've created. Let's face it, they are waging a war on the Internet first.
Edit: Disclaimer: I know that my post is quite biased, especially this one. I'm not suggesting that people should wage a war to Chinese government. Please take my words just as a (biased?) sample from an ordinary Chinese citizen who is really tired of the government's censorship.
The two most important aspects in war are casus belli and plausible deniability. China has the latter and github lacks the former. Thus github would lose by default in any 'war' against the Chinese government.
It just says given this URL, we return an header that tells the browser to redirect. The only thing faster would be just dropping the connection as soon as its given to you.
> redirect the workload to some Chinese government's website and let them suffer what they've created.
I'm not saying this is a great idea, just that something could be done.
Edit: I think is the dataType: "script" part. From jquery docs:
> "script": Evaluates the response as JavaScript and returns it as plain text.
For quick and dirty I like it. Its not exactly long term or really destructive - but its kind of a cute and clever attack.
Github's response to pop an alert was priceless. Sure it probably annoyed the hell out of millions of Chinese people - and their government will probably claim Github attacked them --- but the truth will out... maybe.
Totalitarian regimes are shady as hell.
Some people have used eval() to do JSON parsing because JSON is a subset of JS, but if the user has any control into making malformed JSON, they could do so to create JS that can do anything the page can from the context of another user, otherwise known as Cross-Site-Scripting (XSS).
Would it have been prevented if Baidu served the .js files only over https? Are there any reasons of using http for anything that Baidu serves?
0. http://googleonlinesecurity.blogspot.com/2015/03/maintaining...
Also on the other link I have seen another relevant article [0] on how BitTorrent could be used for attacks from China.
Scary stuff.
Google analytics is also served both over http & https ? Can anyone shine a light on that ?
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
This is just my theory: I think that GFW is currently entering its next stage, which probably includes MITM attack to TLS traffic and some attack specific to websites outside China. I suppose that since everything now is in a "research" stage, they are just trying to see if the technique works and how much it could go.
Disclaimer: I was an user inside China and being blocked from the real Internet. So please take my words with a grain of salt.
Everybody's talking about how this is a targeted attack against GH, but I'm starting to think you might just have hit the nail on the head...
Since we all know that the Chinese government would never do a thing like this, it must mean that there is a very powerful hacker group behind this. And they are probably DDoSing for profit. Who knows what they may do for profit next? Spy on users? Steal passwords? Credit cards? Impersonating users?
Until Baidu implements a secure crypto solution that can prevent this malevolent hacker gang from sending corrupted scripts, it would be very irresponsible to use baidu analytics!
Why?
Shoe's on the other foot now, hahaha!
:)
https://gist.github.com/zhufenggood/7bb040b1effb71d14bcc
Here is deobfuscate version using http://jsbeautifier.org/
https://gist.github.com/zhufenggood/6a38c2a2b2185977b3cb
Github notice that, it replace that DDOS http request respond with a alert("WARNING: malicious javascript detected on this domain"). That is why some Chinese guy gets a weird pop-up with English text when visiting Chinese websites.
But thinking about it, there are fails on so many levels.
Even though I don't think this would have proactively helped this DDoS, but it can't be closer to the truth.
CDNs hosting libraries / fonts / resources for the web are going to be targeted more and more, it is just too attractive to malicious people.
(Obviously if the attack is against, say, Chinese users, and your site's legitimate users are mainly in Estonia, you can do filtering, or if the attack only hits an obscure URL, but the attack doesn't have to be weak in that way.)
There are a bunch of potential ways to address it, but they all work best if you have a site with a defined community of users. If you're a large public site, without login, it's hard. Some of the better techniques are in-browser challenges (JS, CAPTCHA, etc.), but it's conceivable with enough endpoint with real browsers and real humans on them, these could get defeated.
>alert("WARNING: malicious javascript detected on this domain")
They can probably serve that without hitting their database servers.
They're using git, don't they? Obviously you miss the web interface to issues, pull requests, etc; but if you don't have git repositories distributed in your own infrastructure you're doing it wrong.
You would if you designed your build process that way. I think it's a good idea to eliminate all third party build-time dependencies. In practice this means keeping a Git clone of everything to use for official builds, or anything that can't risk a third-party being unavailable when you need it most.
I mean, having a proper local master backup is a great idea, but you could survive without it in a case like this. Just set up a new repository somewhere everyone can get to it and have everyone push to it.
https://github.com/toolmantim/bananajour is pretty nifty.
I've worked at places with enormous enterprisey SVN checkouts that might take an hour or two to slowly download off the internet even though the developer at the next desk may have the exact same bunch of bits available without having to go all the way out to the internet and back. Just solving that problem with better Git tooling could mean a better experience than the current re-centralised decentralised version control...
I don't think there are many companies who have the depth of info security knowledge that Github can draw upon. You might believe that running your own server and locking it down to your local network (as best your team can do that) is better, but I'd rather trust Github even if that means my code is available online. While the chance of being victim to a non-specific attack is far higher (Github is a much bigger target; I'm affected if they're attacked), the chance of someone targeting my code and actually getting it are far, far lower because Github is better at making things secure than I am, and they have people who are paid to make sure things stay that way.
A policy of only having your code on Github has it's flaws but making sure it's secure isn't one of them.
No, it doesn't. Why should it?
That's true and I believe in that as well. However, GitHub also poses a much larger attack surface than any single company and it's safe to assume that once someone is in, they're going to get _everything_. Apart of that, you're also vulnerable to having GitHub disgruntled employees accessing your data, and the inherent vulnerability in having a remotely accessible repo in the first place.
And our Internal security team (BT Security) was bad news if you where investigated - they have a ferocious reputation
http://www.wired.com/2010/09/google-spy/
>Google acknowledged Wednesday that two employees have been terminated after being caught in separate incidents allegedly spying on user e-mails and chats.
>David Barksdale, 27, was fired in July after he reportedly accessed the communications of at least four minors with Google accounts, spying on Google Voice call logs, chat transcripts and contact lists
>In the case of one 15-year-old boy Barksdale met through a technology group in Seattle, Washington, he allegedly tapped into the boy’s Google Voice call logs after the boy refused to tell him the name of his new girlfriend. Barksdale then reportedly taunted the boy with threats to call the girl.
>Barksdale also allegedly accessed contact lists and chat transcripts of account holders and, after one teen blocked him from his Gtalk buddy list, reversed the block. A source told Gawker that Barksdale’s intent didn’t appear to be to prey on minors for sexual purposes, but simply to goad them and impress them with his level of access and power.
I recall at least two vulnerabilities that GitHub was exposed to. The mass-assignment one from Rails (which they didn't fix until after they were the poster-child for it), and the cross-site one, which prompted them to use github.io.
I suspect their security teams are better now but you're still taking it on faith. By using 3rd-party services, you are increasing your exposure, not diminishing it. Someone who wants to attack you specifically will do so regardless.
The problem is not having a third-party host, it's having no redundancy.
We use this, in our capistrano tasks :
set :repository, ( ENV[ 'FALLBACK' ] ? fallback_url : github_url )
This post does not specify what happen on a `git fetch all`. I suppose it takes the first one, and you have to specify other remote entries if you want to pinpoint the remote repos to use ?
Note that the deploy I mention will do a fetch, so this solution does not replace the fallback variable.
Past that I've not used it that often outside of having a cheap cluster solution in a way. Try it out and let me know!
We do not use github only for issues and PRs, though, but also with repos hooks : pushing to a branch trigger a build on our CI, which then deploy our staging env if successful.
Of course, we could do as well with git hooks and writing code to do the web push. I just can't see any reason to migrate to that.
I guess I'll have to look into what ElasticBeanstalk is doing.
Our internal source control server is more reliable than our local shared ethernet with cable backup combined with the reliability of github.
We've had 43 minutes (scheduled and unscheduled combined) downtime in the last 11 years across three VCS technologies hosting internally.
If it was github and remote, we're talking about 7 days of downtime in the last 2 years alone.
We're definitely in charge and it has consumed virtually no admin effort in that time. The VCS server currently has an uptime of 3 years, 5 days, 20 hours and 38 minutes. It's also shifting 10mbit peaking at 200mbits pretty much all day every day 24/7/365.
(CentOS 6, SVN, HP DL380 for ref)
To be fair we haven't patched that box's kernel since it was rebooted but we really don't have to bounce kit very often. Most updates are handled through service restarts. Only updates that are an attack vector internally are actually applied too.
As for the usual problems of power, we have battery and generator backed support. Redundant NICs on two VLANs and redundant power supplies on two feeds and a RAID array obviously.
We also have a large memcache deployment that hasn't been restarted in that timescale as well. Even the memcache processes are that old. Works wonderfully.
Also CentOS is pretty damn stable.
If this was public facing it would be a slightly different story as we'd be patching the kernel too.
(1): by uptime I mean time up during office hours under the assumption that dev workstations can run. There's no point in measuring uptime during a power-outage...
but for small projects or quick deploys absolutely just go ahead and commit the modules.
The bower documentation actually advises checking bower_components into source control.
Another thing I used to do was have a simple script that tried to do npm install, if that failed, npm install using the NPM Europe mirror - although I see now that that one's been deprecated because NPM's primary repository has become more reliable and hosted at a CDN with multiple locations across the world, etc.
You're also insulated from npm and GitHub downtime - win-win-win.
The build step is where you install deps, do compilation, etc. Then it saves the output. The deploy step can just copy all the files from the last known good build to the new/updated servers.
This way you can always deploy a last-known-good release quickly and without external dependencies.
It also lets you take advantage of package caching on the build server so that as long as your deps don't change, you can deploy new releases to old or new servers without hitting any external services.
Major ISPs from the west should definitely consider blackholing all traffic coming from there to avoid DoS attacks, spam, etc. – From my experience, this would mitigate spam by a 50% or even more.
Or the Americans? How about the Scottish? I know I personally want rid of the Irish from the internet, so we should black-hole all traffic from there too while we're at it.
/s
[1] http://en.wikipedia.org/wiki/List_of_countries_and_dependenc...
Props to China. 很好!
Edit: works again now