GitHub: October 21 Incident Report

It's a tradeoff, just like anything else.

We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it.

Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.

>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over,

Welcome to the SAAS world. Where you offload institutional knowledge and hiring to a 3rd party and roll the dice.

This is why we built and manage our own git servers internally. We run emergency services infrastructure so we can not afford to be disconnected from our version control. Ever.

Same reasons we don't run our application and supporting infrastructure in a cloud provider really - apart from the complexity of our particular network which has a heavy dependency on mobile and satellite links - we cannot guarantee our customer of our availability if we don't control the compute, storage, SAN fabric, and as much of the network infrastructure as is possible.

Why would state actor level attacker target for being a code hosing platform? At best they can temporarily suspend operations.

> very up-to-date and informative

Is that satire? It said 2-hour ETA 5 hours ago and the last update was over two hours ago.

Yeah but their recovery estimates were completely off. Pulls, hooks, checks and issues still are unusable

Every company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.

Which brings into question... what is so special about the "front-end" features that Git provides? Why not use another third-party service that integrates into your Github/Gitlab/Bitbucket repo, storing the meta data of all the pull requests, code reviews, discussions, etc, inside a "meta repo" itself? I see this incident as an important note in taking steps to decouple important "meta" features from the cloud-host itself. It will be better to keep the cloud hosts as a bare bones skeleton that we can swap in/out at will (Github, Gitlab, etc), and then use another layer on top to provide all the nice fancy features.

> this disastrous calamity by Github really made me try out Gitlab and in the process

Every company faces problems like this. GitLab infamously lost their entire production database [0] -- and I think we can agree that's more serious. Knee-jerk reactions to incidents will leave you without any "trusted" services, because mistakes happen to everybody at some point. BitBucket has certainly had its own fair share of downtime.

[0]: https://about.gitlab.com/2017/02/10/postmortem-of-database-o...

Based on what I read on their engineering blog they will eventually have a similar outage.

> Actually, I don't know why I pay for Github private repo's.. I might as well set-up two origins, one at Gitlab and one at Bitbucket, for all my privates. Then keep Github as a public front-facing portal.

You do realize that every cloned tree can be a git "repo", regardless of the machine it's on, right? GH and GL surround the repo with some other things (bug reports etc) to rope you in but if you are cloning from one to the other you already aren't migrating that stuff as well, so it's not really clear what additional value that provides.

Is there something I'm not seeing?

It's not that bad. X definitely modifies Y but it may also modify Z. You are informing the audience of this fact and it provides them with 'warm fuzzies'.

Happened to me already. Created a new branch but it kept me informing me the non-existence, retried a few times and shortly after was created with a bunch of duplicates.

"Partition failure" is a relatively specific term of art that implies it was a partial database failure, so "stuff broke because we lost part of a database".

It may not be in GitHub's best interest to describe much more than that because which database, which tables, and how they were partitioned can be secret performance sauce.

Postgres partitioning: https://www.postgresql.org/docs/9.1/static/ddl-partitioning.... Redis partitioning: https://redis.io/topics/partitioning

I suspect it's just that you hear about them all

It’s hard.

But really, do they have that many problems?

Define regularly.

I can recall only 2 incidents this year. I think that's not too bad considering the level of traffic they have to contend with.

Lots of users. Also, a lot of package managers rely on hosting platforms like GitHub to host their packages, so if Github breaks, a lot of CI processes around the world break.

> Is there anything special about it?

Yup. You notice when they're down. Whereas when your self-hosted git server goes offline for a couple of hours, nobody else notices.

It's just noticeable when it happens during someone's work day.

No, that's what a typical single organization with a typical RDBMS-centric non-resilient architecture can provide. But of course it's very hard for organizations at certain sizes to do it better, it's something they have to start with.

What's special about it is that you have terabytes of data to keep available, and at scale git does not play well with technologies like NFS or cloud object storage like S3, so each major provider either pays a lot of money to specialized vendors or has homegrown solutions to deal with the problem.

So on top of your usual problems with keeping a cloud service up and running, you also have that git IO problem to contend with, and to rub salt in the wound, that wrinkle also makes it difficult to fully adopt many "standard" cloud architectures or vendors (such as AWS) which work for non-IO-heavy applications: you always have this major part of your infrastructure that has this special requirement holding you back at least partially (and that can hurt your availability for related services which are not even IO-heavy).

(That said, it's hard to guess whether that was the problem, a contributing factor, or unrelated entirely based on the details provided here.)

source: I work at Atlassian (though not on the Bitbucket team) and occasionally chat to current and former Bitbucket devs on this topic.

Sure, but I don't think they have the time since the outage is still ongoing.

I created a new repository at about 9 UTC, and it started to work some of the time at 12 UTC — I've pushed the code from my computer, and pulled it from elsewhere.

However, it's still intermittently failing as of 14 UTC, so I haven't managed a Maven release build yet.

I created a new repository 30 minutes ago.

As I initialized it with a readme and an ignore file, I had to clone it. Cloning only succeeded by doing `watch git clone` and waiting a few minutes. But it worked.

Gitlab has had its own issues.

That's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.

Average 50 percent active (as in daily usage)? 2 hours lost?

I'd estimate a couple of order of magnitudes less...

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project dependencies hosted on Github to do the initial build of my project, which doesn't happen very often.

In order to objectively assess their level of professionalism, you'd need to know what happened, and what's going on in there right now. Think about it this way - can someone break things where you work right now to cause an outage of this magnitude? For all places that I've worked the answer would be "yes, of course".

You need to judge them by the recovery in the context of what happened. GitHub has a world class engineering team, but major outages still happen. Even in the context of a major outage, they have best-in-class status pages and frequent around-the-clock updates.

I helped manage a hosted DVCS and CI system in my previous job, and do you know what we would've called an outage that happened at 7:00PM and had recovered partially before start of business?

A Tuesday.

Wait for the RCA to come out before throwing any stones.