GitHub had connectivity problems | Hacker News Reader

GitHub had connectivity problems

status.github.com

197 pointswebmonkeyuk162 comments

Does anyone else find the Github status "messages" page [0] a bit jarring in how it's organised? The way that, when read from top to bottom, time goes forwards within a day but backwards across days?

I guess I normally wouldn't notice, but there are currently some messages on there from today and yesterday and I found it hard to read as a story (either forwards or backwards) - I had to jump around a bit to figure it out.

[0] https://status.github.com/messages

I just noticed as well, i got really confused for a few minutes. Bit of a weird design.

Design looks very, very confusing regarding timestamps. https://i.imgur.com/wminZWg.png

It probably uses the local time zone name (for me it shows CEST) and was written by someone who assumed all time zone names are always abbreviated.

I also live in CEST.

Is there a repo you can make an issue / PR on?

Doubtful, GitHub isn't open source software.

I was going to comment the same when I saw yours ahah.

It's fixed now ^_^

It seems a certainty to me that github will be breached one of these days, and all internal data (i.e., private repos) made public. On that day, so many companies will inadvertently become open source!

Do we have any info on what steps github takes to prevent this? I ask as a paying customer (with both personal and corporate accounts).

If such a massive scale breach would happen, it would probably have to take weeks because GitHub probably has so much data. (I would guess on the order of hundreds of TB or a few PB).

It would be more likely to just have handful of high visibility repos.

You're probably right. It's unlikely that a hacker would care about my private repos.

That being said, I'm also curious about GitHub's efforts to prevent such a scenario.

Which makes me wonder: how would one prevent such a scenario, where you cannot simply encrypt with a shared key given github's auth model ?

Like http://homakov.blogspot.nl/2013/03/hacking-github-with-webki... or http://homakov.blogspot.nl/2014/02/how-i-hacked-github-again... ? ;)

You would probably get a better answer if you asked Github this question.

Github employees are here on HN, and just as likely to answer here vs a tweet or whatever.

Yes, but this isn't a support channel. Twitter and their support email(s) are a support channel. They may answer the question here, just like they may answer the question if you sent them a snail mail with it written in crayon. But they are not required too.

Support channel is where I go when I can't access my account, or want to submit a feature request. Anyway, it'd be awesome if someone posted a technical response here, but I don't expect them to.

On that day, so many companies will inadvertently become open source!

Well, no. They would be inadvertently source-available. If the license doesn't comply with the Open Source Definition, it's not open source.

EDIT: Downvoted as usual for correcting false impressions about how free software and open source works.

This. If we accept "stolen and leaked information" as open source, we may as well start coaching discussions about privacy leaks in the same way. "Ashley Madison customers' information open sourced"

"EDIT: Downvoted as usual for correcting false impressions about how free software and open source works."

Are you seriously suggesting that the OP you responded to was comparing "stolen" or "leaked" software to open source? That is probably why you're being downvoted, not because people around here have a false impression as to what constitutes open-source.

"Open source" has an established meaning. Even if the poster in question was using the term facetiously, I feel it must be corrected since there is legitimately a large body of misconception surrounding the mechanics of free software and open source, and such humor may fuel it further.

Moreover, it reflects a critical flaw in the open source dogma as compared to free software. Open source puts source code at the forefront, which is fallacious. The key elements are the ability to run unfettered, study, modify and redistribute identical or modified versions. The source code is a necessary precondition for properly exercising those freedoms, but not a central focus in any real sense. It is easy to misinterpret "open source" as being about publicly viewable source code, and many companies have exploited this to their advantage presently or in the past (GitHub with Atom, Epic Games with UE4, etc.)

> On that day, so many companies will inadvertently become open source!

The OP was explicitly calling such stolen or leaked software open source. Please read more closely.

I did not downvote you, but I think you're being downvoted for lacking a sense of humor, not for correcting "false" impressions.

I'd imagine that for legal/security reasons, you'd get about the same response as any other company:

"We use better than industry standard encryption on private repo data, encrypted with AES-1024 and encrypted in transit via TLS-3 etc etc..."

But, I don't know. What was that quote? "Don't put anything on the internet that you wouldn't want to see on your grandma's coffee table."

I don't think most people would mind having their source code on their Grandma's table. Free off site backups :)

Touche. You'll be the first invited to my new BGCTaaS business then—Backup to Grandma's Coffee Table as a Service.

I wonder if this is related to https://news.ycombinator.com/item?id=10101469

I'm nobody to assure this but I can see one thing common in both the cases, 'Dragon'.

All these ideas like "let's fetch our deps directly from github" sound good until github is not down.

Yeh it's definitely preferable to have one of the two developers your small startup can probably afford spending a good portion of their time rolling out, securing and maintaining your own infrastructure

Yeh cos those are two only two options available. Good thinking.

GitHub is not meant for distributing dependencies. Maven Central on the other hand is, the difference being that it is mirrored and if repo1.maven.org goes down, it's not a big deal and your project can still be built and deployed.

.. until they become a target of a DDoS.

Also, if GitHub is down you can still fetch your dependencies from somewhere else.

You have not understood the concept of 'mirrored'.

Well I guess all mirrors could get targeted. Also, what stops github from getting mirrors themselves?

An internal GitLab install isn't that time consuming to maintain. Also: dat false dichotomy.

False dilemma much? Private versions of most dependency repos exist (npm, maven, etc)

You have to fetch them from somewhere. If you are fetching deps from multiple sources, there is a much higher chance of at least one source being down.

You can fetch them. Once, then cache them

As well you could if fetching from Github?

Not being able to fetch one is just as bad as not being able to fetch all.

That was my point.

That applies to an in-house solution as well.

Inhouse DDoS seems less likely though, at least ;)

Sure, but I didn't specifically mean DDoS attacks, more of the human error kind. People tend to be more trigger happy with changes when it's "only" an internal system.

Or "let's move our entire business workflow to github!". Doh! This is why you should run indefero/srchub/gitlab or any of the other self hosted source code control apps.

I would argue from personal experience there's almost the same (or higher) likelihood of internal sources going down for other reasons since most people's IT departments aren't as efficient as GitHub's.

For most, the cost of setting up and maintaining those systems than they lose when Github is down.

It all depends on your infrastructure.

I once worked in a group that used SVN for SCM, mediawiki for wiki and Trac for tickets. I voiced my opinion about that setup many times but management felt that was the most efficient way to work...

Products like indefero/srchub have all of that from a single interface and it's very easy to maintain. I don't know how easy gitlab is to install/maintain - perhaps sytse can pop in and go into detail about that. Assuming the system works the maintenance would be minimal.

You called me :) Anyway, installing GitLab should take under 5 minutes on a fresh Ubuntu 14.04 server with https://about.gitlab.com/downloads/

And then you can set a daily nightly cron for: sudo apt-get update sudo apt-get install gitlab-ce

Don't forget to take snapshots of your server for backups.

> And then you can set a daily nightly cron for: sudo apt-get update sudo apt-get install gitlab-ce

AHHHH!!!! Don't recommend doing that. Only those who are crazy do that. Imagine waking up in the morning and your SCM system is broken due to an update that was automatically applied - I wouldn't want a sysadmin (or developer) to see that before their first cup of coffee.

When you update - does it automatically apply migrations or other database upgrades that might be necessary?

For the reason mentioned it is better to set the timing during the day (that is what we do). But if you upgrade automatically during the day you'll have a bit of downtime during the day.

By default the Omnibus packages will stop, run migrations, and start again, no matter how “big” or “small” the upgrade is. The behaviour can be changed by adding a /etc/gitlab/skip-auto-migrations file.

Obviously if you installation grows you can review the needed upgrades in advance and prevent downtime in most cases.

This was a controversial/contentious statement during the last DDOS attack. I wonder if any minds have changed.

I'd venture to say that more downtime occurs via local issues (Internet, hardware, virus) than Github.

I do wonder why GitHub gets hit, and people say it's a high profile target, yes almost definitely.

Another thought occurs to me though, I guess one of the problems when writing a DDoS is measuring how well it performs. The GitHub status screen provides a lot of useful metrics for tuning such an attack.

If I may posit...

1. Github is very well known and cater specifically for the tech crowd. So, an attack on Github is more likely to be talked in the tech crowd, which I would assume the people who would try out DDoS attacks be more likely to be part of. Validation is a weird thing.

2. In a convoluted sense, taking down Github doesn't harm as many bystanders. If that makes any sense...

3. As you've mentioned, very clear useful metrics for the attacks.

4. Crowding effect? Maybe attacking Github has become some 'cool' things to try in that community. I'm just imagining at this point.

> Validation is a weird thing.

And bragging rights. Imagine the street cred you get amongst that community if you take down something like github...

>I do wonder why GitHub gets hit, and people say it's a high profile target, yes almost definitely.

It could be related to things like this: https://news.ycombinator.com/item?id=10101469

Possibly just Round 2 of the massive DDOS from March:

https://github.com/blog/1981-large-scale-ddos-attack-on-gith...

http://arstechnica.com/security/2015/04/ddos-attacks-that-cr...

I'm getting 500 timeouts on Bitbucket. Didn't Google Code go read-only today? Don't you guys think Bitbucket and Github might be having issues due to people migrating off of Google Code at the last minute?

Possibly, but they are having degraded API performance and high load on their front-end systems.

http://status.bitbucket.org

Totally speculation, but maybe related: https://news.ycombinator.com/item?id=10115641

Yeah my first reaction was it is government based and probably China.

I know, we should keep building up their economy by manufacturing nearly everything there, that should stop them.

Over the long term, it will. An educated wealthy populace doesn't accept tyranny for long.

To some extent, most likely, yes. But my perspective on the PRC is that the powers that be know that they will only remain in power for as long as they can provide, or rather, the illusion of them providing, a continuous increase in wealth. This in part would be why information is key in China. Assange had a very positive view of it when he said "I often say that censorship is always cause for celebration. It is always an opportunity, because it reveals fear of reform. It means that the power position is so weak that you have got to care about what people think." [1].

[1]: https://wikileaks.org/Transcript-Meeting-Assange-Schmidt#996

You know, the implication of that for the U.S. is pretty disturbing.

The US isn't big on censorship, they're big on data collection. The MPAA and RIAA are HUGE on censorship, but they aren't government entities.

Except they don't know it is tyranny, just like Russia, they rely heavily on subtle propaganda to manipulate their population.

Attribution dice need to be rolled for this one (https://twitter.com/AttributionDice)

I'm wondering if there is a corresponding traffic spike on the hn servers ;-)

And a famous break-through-GFW software called goagent is deleted from github. *sigh

https://github.com/goagent/goagent

You may want to explain how this is related?

Well that's me not able to work.

If only git were a distributed version control

It would be nice not to have to rely on centralized servers at all. A P2P/torrent client built specifically for sharing code from git repositories, perhaps?

But there is much less commercial incentive behind a p2p alternative. Centralized services are easier to monetize.

I look forward to the day governments start to fund free software, for example through the GNU project. Then we'll all have better tools.

Actually, there were already some fundings. Germany funded GnuPG to port it to Windows.

What do I care what is the new world record in sprinting? But I very much do care about a new release of libreboot, libreCMC/openWRT or Debian!

I hope one day we will see peaceful international competitions between nations of who can provide the best/most used free software.

Hear, hear. I've talked about this with people and it's surprisingly hard to convince them that government-sponsored software is a good idea. While a tiny fraction of the amount we already spend on commercial software would utterly transform the free software landscape, it's not politically viable as long as commercial software vendors have lobby power.

Let's not forget that it is not in many governments' interest to have distributed tools (with strong encryption). It is much easier to take down stuff on centralized services.

BTW, it is hear, hear ;).

But of your government is at odds with an oppressive government, it would make sense for it to fund that kind of software so the people could take it down, or at least make trouble, themselves.

Thank you for the correction :)

> it's surprisingly hard to convince them that government-sponsored software is a good idea

That's because it's not. Governments are not reliably good, and their money comes with far more strings than private money.

The FSF have it right: people should write free software because proprietary software is immoral.

> [Government] money comes with far more strings than private money.

Have you ever worked on a government-funded project? The Tor folks have and do. :)

> ...[P]eople should write free software because proprietary software is immoral.

USGov does fund Libre and Open Source software. One big example is the Tor Project.

Instead of gov. funded software, I like the idea of hedge fund backed FOSS, where the commercial software competition are publicly traded. This would allow the hedge funds to realize returns, but only if the FOSS produced is viable. Such an arrangement would probably benefit under a non-profit foundation to administer the project(s).

I'm just back from lunch, so I'm struggling to follow. How would the "hedge fund backed FOSS, where the commercial software competition are publicly traded" work?

The government is funding free software. Just not in the way you might expect. :-)

https://www.youtube.com/watch?v=3jQoAYRKqhg

Free distributed P2P software makes technopolies like Github AND mafiaficialities like nationstates obsolete. We do not need to and should not rely on the government to fund that.

There is this https://code.google.com/p/gittorrent/ although I've never tried it...

edit: a more recent (?) link https://github.com/cjb/gittorrent

It would be really nice to get it to succeed. Decentralisation is almost always preferable.

I think they're two different projects, but this is exactly what I thought of. A separate Chinese DDoS was even mentioned by cjb in his announcement of GitTorrent a few months ago: http://blog.printf.net/articles/2015/05/29/announcing-gittor...

See http://ipfs.io and https://github.com/cryptix/git-remote-ipfs

IPFS, anyone?

Yeah, I'm hyped too.

> A P2P/torrent client built specifically for sharing code from git repositories, perhaps?

Or, we could both set up HTTP servers on our respective machines and simply push to each others repos.

I thought that was the point of git...

github is not only git... how many times will we have to tell that?

PRs, wikis, issues are the reasons why we use github...

If only it were possible to keep those things in git itself. Oh wait, it is.

Can you explain? How do you store issues in git?

> How do you store issues in git?

An org-mode file named 'issues,' with each issue under its own heading?

A directory named 'issues,' with an org-mode or CommonMark file for each issue?

A directory named 'issues,' with a directory for each issue, with CommonMark, org-mode or restructured text files for each person's comments?

You commit your issue database to the same repo as your code and track it alongside your code. Then you just need a tool to manage the database for you, and there's a few tools out there to do this:

http://ditz.rubyforge.org/ http://pitz.tplus1.com/ https://github.com/jeffWelling/ticgit http://www.bugseverywhere.org/

... and probably others. That's just based on a quick Google search.

One nice thing about this approach is that your issue's state follows your code through merges. When you fix the bug you mark it as 'fixed' and commit that change to a branch. When it gets merged into master, the 'fixed' status gets merged as well.

One can create a fully offline version of github fork using https://github.com/joeyh/github-backup + https://gist.github.com/piscisaureus/3342247 ("Checkout github pull requests locally")

(PS: why cannot I reply to paulrouget's message?)

Oh cool, I didn't realize git being distributed solved the communication problems that arise on an engineering team.

Back to work everyone, git is distributed version control so we don't need pull requests or gists or any kind of easy-to-read historical record that is also accessible for non-engineers.

If it's that critical then someone needs to fix the SPOF.

If it's really that important, why not host your own gitlab or pay for a self hosted github installation?

If only there was a way to do p2p communication that you could use to send patch files. Uhmmm.

I think the bigger point is to have a contingency plan in place so your whole engineering team isn't sitting on their thumbs with stupid grins on their faces.

You don't have to edit the files directly on github in the browser, really ;) Just edit locally, commit even! It's magic!

http://stackoverflow.com/questions/11459475/should-i-check-i...

That might not be your specific problem, but I'm guessing a few people have this problem today. I'm glad I check my libraries into my local repository :) Using a local fall-back for popular libraries hosted on cdns is a good idea too.

To be fair, I don't do it because I think it's stupid not to, I do it because I often work on my laptop while travelling and have to be able to continue to work without an internet connection :)

Ugh, how horrible! DDoS'ing github is like kicking a puppy in the face.

Github has certainly done an amazing PR job if that is a common perception.

Isn't it more like barricading bus drivers inside their homes because someone you don't like takes the bus to work?

I always say that making an argument using analogies is like trying to tie your shoes with laces made of butter. :)

https://pulse.turbobytes.com/results/55dc40efecbe400bf800146... problems during TCP connect

https://pulse.turbobytes.com/results/55dc40faecbe400bf800146... traceroute looks ok..

I wonder why are they providing time in FLE[S]T (same as EE[S]T)? Are their engineering team based in Eastern Europe or what?

No, you are.

They're trying to detect your timezone and show the local time. Nice of them, however additional UTC for vpn users and lines with broken geoip would be also good.

If GitHub folks are technical in nature, couldn't they simply have a secondary mirror of their own that they host on their own servers / clouds and then reference both primary and secondary? Perhaps redirect the secondary to the primary if it is reachable to avoid bandwidth issues?

About 3 hours in github twitter profile: https://twitter.com/githubstatus/status/636159212876361728

Bitbucket is currently experiencing some issues too [0]. Is it related?

[0] https://bitbucket.statuspage.io/

I'm going to assume yes. And I'm going to assume what ever is the issue has a mirror on bitbucket.

But I don't think anyone can say for sure.

Guys, what's the purpose to attack a service like Github?!

It could be for something like this: http://arstechnica.com/security/2015/04/ddos-attacks-that-cr...

Institutions that don't like code that is hosted on / distributed via Github.

Proof that you can take down a service as large and as well connected a Github perhaps.

What makes you think Github is "large and well connected"? Don't get me wrong, I love GitHub, but due to their architecture decisions, they aren't anywhere near a great example of best practices in the industry for preventing DDoS attacks.

Maybe they're unable to take down a service larger or better connected than Github, perhaps?

I'm intrigued. What architectural decisions?

One that would really help with DDoS attacks is if they didn't do everything on github.com. Layer 3 DDoS mitigation will always be cheaper and more effective than Layer 7. If it's not obvious what I mean, here's an example:

Current: http://github.com/jsjohnst/project

One (of many) better approaches: http://jsjohnst.github.com/project

Citation needed.

China does not like some of the projects it hosts, namely the ones used to circumvent its Great Firewall.

That's a clear signal those projects have value and that we should work to make more like them.

At some point we should just kick China of the net, just for a week or so.

If you really want the Chinese people to protest, try cutting them of from the Internet for a few weeks. Let's see if that won't work.

What do they possibly get by DDoS'ng GitHub ? Is this out of pure malice or are there any probable commercial gains to this?

There's two common flavors of comment on this post, namely "why would anyone do this?" and "well, I guess that means I can't do any work this morning." These could go together somehow. :)

Indeed.

Considering its the Chinese government, yet again, its probably a mix of both. It makes it harder for their nationals to get VPN software that goes through their firewall and also is an assholic statement against the West's ideas about freedom of speech, assembly, and peaceful changes of power via popular elections and multi-party government. The Chinese are most likely hoping they can force github to get rid of these projects by threatening periodic DDOSing.

They also probably feel emasculated that South Korea told their pet regime to knock off the stupidity and China and North Korea capitulated to their demands yet again. The timing of this is far from a coincidence. China's foreign policy is almost 100% hinged on having North Korea attack the west with impunity and they don't like that Park is actually pushing back against this dynamic. So now China is having its hissy fit on Github instead on the Korean peninsula.

Yeah, lets keep rewarding them with factory contracts, right guys?

Someone doesn't wanna work today.

Mandrill reported they were investigating system slowness a short time ago, also.

GitHub DDoS, pointing out ironic single points of failure since the beginning!

Is this because of shadowsocks and China is pissed again?

China!

I haven't experience any connectivity issues on Github today. Just pushed some code and it worked fine and promptly.

I've always wondered why github hasn't considered switching to a distributed sub domain layout to help ameliorate this problem. Surely if they spread their infra out that would make ddos that much harder. A subdomain per user or repo should work wonderfully

Possibly for SEO reasons.

I don't think that will solve the problem.

Well if x% of repos remain up that goes a long way. Right now it's all or nothing.

Also if you're going to respond include a reason why you disagree

Separating users by subdomain leaks information. An observer can trivially see your DNS queries but not which repository you access over https or ssh.

There is also little benefit in it. If the subdomains only point to the same servers then the same level of traffic will still take them all down, but if different subdomains point to different servers then it makes attacks easier because the attacker only needs enough resources to overwhelm 5% of the servers instead of all of them.

Title should be "Github _is_ under DDoS and having connectivity problems"

edit: TIL my English wasn't as good as I thought.

That's subjective. In America companies tend to be referred to as singular entities, but in the UK they would be referred to as plural. Neither is wrong, just different idioms.

TIL

As you say - In English they would be referred to as plural. I'm English.

No, 'are' is, I believe, also incorrect in BrE. It's commonplace here, but not correct. (Or so I was taught!)

http://blog.oxforddictionaries.com/2011/09/agreement-over-co...

> In British English it’s absolutely fine to treat most collective nouns as either singular or plural – you can say my husband’s family is very religious or my husband’s family are very religious.

http://itre.cis.upenn.edu/~myl/languagelog/archives/001874.h...

etc.

With something like collective nouns it's probably wrong to make blanket statements about correct and incorrect, even if you're a prescriptivist not descriptivist.

EDIT: And while I love (but am hopeless with) English usage this is the least interesting bit of the submitted article.

Ah but that's different. "Company nouns" are not used as collective, they refer to a singular entity - the registered company, not it as a body of people.

But they are.

https://dictionary.cambridge.org/grammar/british-grammar/nou...

http://www.economist.com/style-guide/singular-or-plural

Note while 'just' a style guide, The Economist - like me - seems (note not 'seem' ;)) not to consider "Tesco" _et al_ to be collective nouns _per se_.

  > A government, a party, a company (whether Tesco or Marks 
  > and Spencer) and a partnership (Skidmore, Owings & 
  > Merrill) are all **it** and take a singular verb.

i'm sorry but... GitHub IS not GitHub are

Examples: GitHub IS a web site. GitHub is a SAAS app.

The only plural aspect to GitHub is its employees and they are not being DDoSed. The web site (singular) is.

I'm not going to bother explaining the problems with "under DDoS" or the other issues with this sentence.

It's a British thing. Don't try to understand it- just let it be. :)

It makes sense, but "is" is too ingrained in my brain to ever go British.

I'd be surprised if no one at GitHub has said "we're being DDoS'd this morning." GitHub is a website. It's also a company of people who probably identify closely with the product they build, like many of us. I don't think it's that crazy to use "are" here.

Perhaps it is this? https://en.wikipedia.org/wiki/Comparison_of_American_and_Bri...

Technically, the Github servers are being attacked.

The "website" (or "app") is the abstract thing that runs on those servers. The servers are trying to service a flood of incoming tcp connections and http requests from rogue clients, which slows them down.

So "are" is not entirely out of place, but of course we all got the message ;).

Reply on news.ycombinator.com