Agreed that's impressive debugging for this issue. But...
> 10 minutes to roll out the fix
That seems very slow to me. 30% of their down time was because their deploy process is slow.
> 10 minutes to roll out the fix
That seems very slow to me. 30% of their down time was because their deploy process is slow.
FWIW here's a write up on their process
Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"
Maybe "deploy" means the "Deploy" section of this article: http://highscalability.com/blog/2014/7/21/stackoverflow-upda...
Seems to target only being able to "deploy 5 times a day". I guess maybe the build time is the limiting factor.
Its just interesting to me the implications of what folks optimize for and that this is considered fast. We have very minimal deploy testing and optimize to be able to revert quickly when there are problems because performance issues like this are very hard to predict. Probably means we create many smaller short hiccups though (that generally are not a full site crash).