Open-Sourcing Our Incident Response Documentation | Hacker News Reader

Open-Sourcing Our Incident Response Documentation

pagerduty.com

182 pointskungfudoi8 comments

Interesting. I ran the support function for some of the world's busiest gambling sites' backends. A lot of this looks very familiar!

Probably the most significant thing I learned was the power of automation through Incident Models. I spent 7 months of my own time, full time, writing them for the previous two years' major incidents. This changed my life, as I simply stopped getting called, and juniors only escalated to me when the docs were faulty.

Any details you can share kind of things you automated and tools that used?

Oh, the automation I'm speaking of there was the automation of human behaviour through following a script/checklist. It helped everyone - the juniors felt more confident about escalating, and they learned from it too.

(I'd read The Checklist Manifesto, which I couldn't recommend enough, BTW).

In more trad automation, I got a little obsessed by automating the construction of environments, but in a human-readable way:

http://ianmiell.github.io/shutit/

and Docker (wrote this book):

https://www.amazon.com/Docker-Practice-Ian-Miell/dp/16172927...