GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble?
If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.
One big recommendation I'd offer is to have a secondary network with as few moving pieces as possible (think a dumb unmanaged Netgear switch or two) to run things like heartbeats and DRBD over. That sort of thing should not be living on the same segment as your production traffic.
All of the major vendors MLAG implementations are limited to two agg switches, assuming you have a pair of 24 ports and a 48 port rack level switch hanging off each, that still puts total servers at less than 1000.
What stops someone from having multiple zones within a datacenter interconnected with suitably large LAGs thereby avoiding being limited by the port capacity of any one device?
Why is the exact number of servers important to the topic, or is it just to satisfy your curiosity? :)
> There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size?
I didn't intend to deflate any nerd egos, just putting things in realistic terms. GitHub is cool. It's the biggest in its industry. It's definitely an interesting problem. But it's still hosted within Rackspace, which in itself isn't that big.
My first recommendation would be setting up a warm-standby or even a hot-standby if possible in another datacenter. It all comes down to state, meaning the data/repos. They are currently using drbd for block level replication of disks in their file servers, this can be done over wan using something like drbd proxy. Another option is to write something in house which replicates write requests to git repos hosted elsewhere. They wrote a collection of services to access git repos on the file servers via rpc, its entirely plausible to extend this. If they are using mysql anywhere thats a pretty simple task for master-slave replication or semi-sync replication in mysql 5.5. Redis also supports replication.
I'm sort of hinting at high availability with read/write but starting out with a read-only solution hosted in another datacenter on the order of 60-120 secs out of sync is not impossible. I think we would all benefit from that kind of continuity right?
Replicate your data to a secondary site, keep it in read-only mode, set low ttl's in dns. Take it a step further and start looking at globally distributed file systems if you are feeling adventurous.
I guess what your saying is in an error making the site entirely read only from another data center. That could be of some use but I would imagine that would require a complete rearchitecture to separate out all writes and also have a method of recombining stuff like logging which will still be written later on.
On the git level though it would be useful for a major use case of github which is to grab code.
My point was really from the standpoint of a system and network wide outage for which it will take multiple hours to recover. Its about business continuity. I've been making comments from the standpoint of a user and the open source community but what about from the other side. I guess its a subscription like service for paid for customers so it has no immediate consequence on revenue but in places I've worked downtime is money lost and customers would actually require us to have SLAs and disaster recovery procedures in place before they would sign contracts with us. Anyway, yea the step from single server to multi server, to multiple datacenters does normally involve some re-architecting at the state layer, everything else is load balancing.
You have to wait a little longer to do your push/pull/merge/etc?
Give them a break.
Another factoid is that Amazon Web Services (AWS) only offers a 99.95% SLA so where did the 99.99% for GitHub come from?
What's really the impact of Facebook downtime? You have to wait a little longer to "poke" your crush's profile/look at their picture/etc? Give them a break.
Rubygems is kind of unusable without Github.
Several ppensource projects documentation is unavailable without Github.
Part of the beauty of git is that it would be absolutely _trivial_ to have a secondary "backup URL" located somewhere else if Github were down. And git would automatically handle the rest.