GitHub experiencing a large DDoS attack
status.github.com
status.github.com
---
We do know that they've been consistently attacked. Github has been unsuccessful in mitigating attacks.
Github needs to explain why these attacks are happening. What is being done to stop them. What has been done to stop them. And if they are different from previous attacks.
- Getting people to use BB instead
- Fake DDOS to get more companies to pay for github:enterprise
- Preventing a system upgrade in a company which has deployment relying on github
There are lots of potential ways to use it...
git daemon --verbose --export-all --base-path=.git --reuseaddr --strict-paths .git/
Can you please elaborate?
You can have multiple remotes for your Git repositories, one of them (Github) being down should not affect your work since you can coordinate a new one with your team in the manner described above.
Here's how to set multiple remotes. (See answer 2). http://stackoverflow.com/questions/849308/pull-push-from-mul...
1. All of this requires that you have more bandwidth than the attacker.
2. ACL drop everything but the ports/protocols you use on your frontend IP. For github.com that would be TCP 80 and 443. (ACLs are cheap, can be done at wire-rate on any capable edge router, so they're a good "first step." In this case it would have the bonus of dropping DNS amplification attacks which are increasingly common.)
3. If it's a TCP SYN flood (probably from spoofed IPs) enable SYN cookies. (Of course now the bottleneck is their load balancer.)
4. If it's an L7 attack, they'll need to identify patterns in the requests to identify and drop the traffic. For example, the attacker may be requesting a single URL only. If it's an L7 attack on a TCP service, they can also automatically add a drop rule for that source address, because at this point the client IP will have been validated (via the SYN challenge).
They may also be able to identify certain patterns in the traffic that can be ACL'd at the edge. For example, IP TTL being identical.
I believe Step 1 is to submit the fact that GitHub is being DDoS'd to Hacker News, reddit, Slashdot and other social networking sites with a direct link to GitHub's web site. This will cause a flood of real people to load the web site, thereby crowding out the DDoS.
The site will still be down, but at least it will be due to real people and not the DDoS.
Anybody has experience with Gitlab and Gitlab CI? How's the flow compared to Github, especially for pull requests, commenting and collaboration?
Does anyone else find it ironic that Git is a distributed version control system, yet we rushed to centralize it and base our entire workflow around GitHub?
Anything business related we use beanstalkapp, for its better price-per-private repo and deployments in any case (no issues with them ever!)
There's also stuff like GitHub Issues, which are separate from the actual Git hosting. So even if the source code is available and the 3rd party tools aren't broken, work can still be interrupted.
Nobody denies your blobs are safe and available elsewhere when Github goes down.
It's the infrastructure around those blobs that makes Github so popular. It's your issues and pull requests, your wiki, your connection to Travis-CI that are gone when Github goes down.
That is why people get angry when it goes down.
You can probably blame those teams for that more than GitHub but people do seem to have bought into the idea that we can just jam everything up to GitHub and get it back later.
The Heroku/AWS outages a few years back had far more of an impact, yet Heroku still (to my knowledge) relies exclusively on AWS-East, because presumably the risk-profile doesn't exceed the threshold to justify the expenditures required to mitigate the risk.
I just wish the attackers would be considerate enough to share the schedule for these attacks ahead of time, so I can plan a longer lunch. The rudeness of these attackers is unparalleled. I'm inclined to write them a strongly worded letter suggesting same.
How technically hard is it to have an internal roundrobin git system that's synced up with the github repos needed, and then redirect to the internal git servers as needed when github goes down? I'm just wondering, no snark.
Github will go down by the nature of it being a web service. Web services aren't 100% by nature.
Then look at something like Gitweb [1], cgit [2], or Gerrit [3] to front the repo. I personally use git, gitolite, and gitweb.
[1] https://git.wiki.kernel.org/index.php/Gitweb
Not a good situation.
Surely by now, GitHub must know who's responsible and putting more in place to mitigate as much as possible before this happens. Right?
Sheesh.
I still very much love GitHub, it just ended up not scaling for us (we have a lot of repositories that seldom needs to be touched, which results in a $20 / month Linode + Backup being a much better solution)
GitLab also allows us to group repositories and gives a little bit more flexibility in regards of git server-hooks. Also, server-side branch locking! (Does anybody know how to lock branches server-side with GitHub?)
(Actually, there are long established yet less high profile alternatives with decent features and better pricing for commercial teams, like beanstalk or unfuddle or codebasehq. No hip cred, but for example supporting archived projects that you can read w/o using up your repo count license.)
The difference in annual cost is significant when you have 20 active independent projects, and hundreds of archived projects, not to mention all the customers needing accounts and access.
I'm reading HN right now because I can't manage pull requests and do code reviews of my distributed team's code. So I'm just spinning in my chair.
That's the so-what.