GitHub was experiencing a DDoS | Hacker News Reader

GitHub was experiencing a DDoS

status.github.com

79 pointstrevorhartman72 comments

Do we really need to be notified every time GitHub or Twitter go down temporarily? It's inconsequential in the grand scheme of things.

Twitter I don't think so but I feel like the majority of people visiting Hack News use Github in some way. Remote Repo, Deploying, Open Source tools that are hosted on Github.

Maybe just me...

Wouldn't that just mean that they're bound to find out that Github is down without needing to see it on the front page of HN?

the might assume that its a problem on their end / on their ISP end.. Clearly enough people think its relevant for it to make its way onto the Homepage via upvotes ;)

Good point.

Is slashdot already notified? nostalgia

Did netcraft confirm it?

I got an error message when trying to connect to the App Store yesterday. The first thing I did was refresh HN to see if there was an outage or issue.

There is a reason there are controls to allow you to upvote, or downvote stories. If you don't like a story, and have nothing to contribute to the story, downvote it. If the story is in the top few, it's because people (that aren't you) are interested in knowing that github is getting attacked.

EVERYTHING is inconsequential in the grand scheme of things.

Edit: Heh, there are no downvotes on HN for stories. My bad.

OT: How much karma do I need to have to see the downvote button?

There is no downvote button on submissions only upvote and flag.

Should there be downvote buttons for submissions?

Maybe? I could go both ways. I think downvoting is important but it's also very easily abused. I like stackexchange's downvoting system where it comes at a cost.

It is, or was, 500 points.

The downvote on users only appeared for me as I pushed past about 650 I believe.

> If you don't like a story, and have nothing to contribute to the story, downvote it.

You should try doing this because I don't think it actually exists.

I do understand why the hackernews crowd is interested in the status of github. What I don't understand is why those same people don't know about status.github.com and @githubstatus already.

If github is a critical piece of your infrastructure, add a couple of bookmarks and never be in the dark again.

Good point on the existence of the down vote functionality, I've amended my comment above.

To your other points though, Github being DDoSed is something I would like to know about, but does not happen so frequently that I should bookmark the status page and check it every hour to see if someone is DDoSing them. That's rather absurd.

Wouldn't you just check it if github seemed to be having problems?

Maybe it's a sign someone should look into an expanded notification system. I imagine someone could cook up an interesting IF case for IFTTT.

A good way for a company like GitHub to do it might be an RSS feed. That won't necessarily allow to define a status message type indicating the occurrence of the problem vs. the resolution - and the severity of the status problem.

The do have the linked website (status.github.com) and a twitter account (@githubstatus) which honestly seem pretty sufficient already.

unfortunately, the status page is fairly inaccurate and delayed.

(note that I'm not agreeing we should post to HN for every blip)

it's not an RSS feed, but:

https://status.github.com/api/messages.json

see: https://status.github.com/api

An outage of either has a measurable impact on productivity.

Chances are good that the attackers are DDOS'ing Github using software consisting of some open source components... hosted for free on Github.

Why would someone DDoS GitHub? Is there some movement against GitHub? Or is it just for fun?

I manage newgrounds.com which gets quite a bit of traffic. We'll get what I think are DDoS attacks at least once a month. I can see our connection tracking stats go up to the millions and traffic spikes way up of course.

I don't know WHY they do this, but last time it happened we got an abuse report saying that we were reported for port scanning from our main firewall / proxy box. Somehow they had reflected traffic off our firewall / proxy to make it try to connect to a bunch of IPs on a known trojan port.

I have no idea how they did this, but it appears that this time around we were being used to scan ports. This is just a stock Debian box with a firewall and port 80 open. Scary.

> I don't know WHY they do this, but last time it happened we got an abuse report saying that we were reported for port scanning from our main firewall / proxy box. Somehow they had reflected traffic off our firewall / proxy to make it try to connect to a bunch of IPs on a known trojan port.

How did you solve it?

Not sure it's solved, but I added some additional firewall rules to block certain types of ICMP packets that they were sending and added some additional logging for when it happens again.

They wouldn't necessarily have received useful data from it. Consider what happens if you spoof the sender IP and port in the first packet of a TCP handshake: the recipient will send a response to the spoofed IP, making it look like they are the bad guys.

Someone isn't filtering Martians properly, or those spoofed packets would have been filtered before they reached you.

Yeah that's what so confusing about this.

I'm filtering martians/bogons, which I see getting blocked constantly.

Burying exploitation attempts in the logs with a flood of api requests?

Could be a shakedown.

Advertisement/testing for their botnet.

Skiddies gon' skid

plot twist.. it's github doing it to github to get bigger companies to pay for github enterprise.

This is your periodic reminder that deploying directly from Github is a bad plan.

Isn't Github having it's annual employee family picnic this week in SF?

This is literally going to ruin someones picnic.

Coincidentally, I just finished setting up Gitlab and Gitlab CI on a private server. If I could set up repo mirroring between it and Github, it would add a layer of protection for getting critical work done (security by distribution?).

Just have both gitlab and github as upstreams and push to both every time. aka in your .git/config:

    [remote "origin"]
      url = https://github.com/somerepo.git
      url = https://gitlab.yourdomain.tld/somerepo.git
      fetch = +refs/heads/*:refs/remotes/origin/*

And nothing to worry about when github is down.

Huh. I wasn't aware you could do that.

What happens if the repos aren't in sync and you pull?

I honestly don't know, should test it and find out. Will look at it later today.

Edit: curiosity got the better of me https://gist.github.com/mitchty/0c2089445c1400d46cb9

Basically exactly what you'd think.

the space-time continuum collapses?

Someone doesn't like GitHub, this has been going on for months.

there are individuals/groups out there who launch a ddos attack against a site, then contact the owners demanding money and they will stop.

Subversion rebels attack again!

They joined force with the Bazaar empire!

Even if there are attacks everyday everywhere, I wonder how/if it can be linked with recent attacks of Apple, Ubuntu forums, and OVH networks.

That's a lot of techies services attacked for such a short time.

EDIT: typo

Who would do something like this?

Russian mafia?

hg extremists!

everyone ping github to show your support

If you are looking to host an internal git server, then you should watch my screencast, where I talk about how to configure a central git server and manage user access with gitolite.

http://sysadmincasts.com/episodes/11-internal-git-server-wit...

I really need to start monitoring GitHub vrs Bitbucket for uptime. I can't think of the last time Bitbucket was down.

It was down when they changed to a new datacenter[1]. I think github is just a much bigger target right now is why it goes down more often.

[1] http://blog.bitbucket.org/2013/07/02/scheduled-downtime-bitb...

We've had some minor issues with Bitbucket being down, or at least inaccessible. Never anything really disconcerting, but it's always a pain when you can't get to your code, even if it's only for a few hours.

Mostly Bitbucket has moments where it seems a little slugish and a pull or push takes what seems like forever.

It was down a few weeks ago, right in the middle of the last big deploy I did. Just had to laugh honestly.

That was scheduled maintenance, however. They had big banners with warnings everywhere on Bitbucket for several days before the event.

I think the problem with BitBucket and GitHub is that i never need to go to the website for most things. All the interfacing I do is thought the Git or mercurial Interface.

Anyone know what they put at the front of their stack? Nginx? Varnish? HAProxy?

There was another one of these, slightly older, that just got deleted...

Also the red banner looks a bit off to me, the "More Status History" just seems plonked in the middle of it.

Github seems to attribute every issue initially to a DDoS

I'm willing to give them the benefit of the doubt and operate under the impression that they know more about their network than you or I do.

Does someone now what "Pages Builds Failure Rate" means on the page? 4.5% Server errors is a bit much...

I think they could be talking about github pages. Considering that's user uploaded content 4.5% seems pretty low.

If there is a hell, anyone who would attack GitHub has an express ticket there.

Looks like everything is still up though. What is being DDOS'd exactly?

Distributed Denial of Service - https://en.wikipedia.org/wiki/Ddos

I don't think that was the question. The question was, I assume: Which services are being ddos'ed?

You must be really evil if you are attacking github!

This is akin to robbing a police convention.

Seems to be working again...

Hopefully didn't type too soon

ah no! plzzzzz hackers don't ddos github! this is just not fair

Reply on news.ycombinator.com