Hate to be the bearer of bad news, but ikawe was joking about the bower thing.
Source: I work at GitHub.
Source: I work at GitHub.
So, you're saying you broke it.
Everybody, over here, gang up on this guy.
> I hope github does make public a official reason tomorrow
I do too, but, mostly I find post-mortems quite interesting to read.
If your services are at some 95% uptime or lower, you're doing something (serveral things) very wrong and its probably not that interesting to me.
Getting from 95% -> 99% you probably did some interesting things there.
Going into multiple .9s beyond that, you're likely doing quite a bit of interesting stuff but what I find more interesting is where you went wrong. Figuring out not just where you went wrong, but, what your incorrect assumptions were and WHY they were wrong. "We believed X could never fail because of Y, and even if X did fail, it would not cause production impact because of Z!"
But now my "technical breakdown info" box has no tidbits in it. :P
I'm glad you're back up now (sortakinda - what sort of traffic are you sustaining right now? :D), but a rough idea of what asploded would be really cool to know about.
Speaking of which, I'd like to take a moment to make a strong point about the fact that disaster-recovery situations don't get blogged about enough. Vague "we fixed it" datapoints get buried in status update logs like it's something to hide and hope nobody brings up.
In situations like these, the only constructive perspective is for everyone to accept that something went horribly wrong and not make a fuss about it, and if such a mentality can be established, this creates an environment within which we can share technical breakdowns of "we found ourselves in XYZ position and then we did these thirty highly specific things in heroically record time to be up and running again", and I think sharing this type of info would potentially be more educational than setup tutorials or the "we switched to X and it improved Y by 1400%" type things the Net's full of. Sure, you'd have to generalize and probably give a lot of backstory about infrastructure, but it's becoming trendy (in a sense) for companies to describe their operations in precisely this way, so it's not completely nonviable.
(Note my use of the angle of "we found ourselves in XYZ position" - maybe a small highlight of what led up to the disaster would be included (worth considering if the information would be educational), maybe not. In a blog context, moderating comments to keep the discussion on-track and constructive may be necessary, but IMO would be worth it.)