Incident Command for DevOps: Learning from the Fire Department [video]

ICS like this is well already well developed for dealing with cyber incidents, e.g. malware incursion or data exfiltration. A really well thought through example is the NIST cyber security framework [0], which defines a cyber defensive lifecycle (identify, protect, detect, respond and recover). At least in the UK, this lifecycle has been adopted by Critical National Infrastructure (CNI) organisations such as electricity generation, transmission and distribution.

The key to successful incident response is to design, agree and test the ICS processes, roles and Command and Control (C2) hierarchies before incidents occur, capturing the results in an Incident Response Plan. The IRP will typically involve standing up an incident response team when an incident reaches a pre-defined severity level. Incident response teams are often structured into bronze, silver and gold levels of command (gold typically including Cxx individuals such as CIO and CFO, not just DevOps roles) that temporarily replace Business as Usual (BAU) management with incident C2. In the UK at least, the bronze/silver/gold hierarchy was developed by the ‘blue-light’ services (police, fire and ambulance), and is itself a simplified version of military C2 chains of command.

If management don't buy into this type of approach, it doesn't node well for an organisation's ability to deal with a crisis.

[0] https://www.nist.gov/cyberframework

For anyone wanting to dig into the ICS the authoritative introduction is the NIMS IS-100[1] course. Be aware that it's your standard government produced course, i.e. not entertaining in any way but generally manages to get the point across.

If I understand/remember correctly, IS-100 and 200 are the required courses on ICS (along with 700, 800 for NIMS itself) for all frontline firefighters on any fire department wanting to receive federal money.

1. https://training.fema.gov/is/courseoverview.aspx?code=IS-100...

Coincidentally am finishing up an EMT class and prepping for the NREMT test. I have been very struck by the similarity between the differential diagnosis process an emt uses and the troubleshooting a server or network issue. Similar problems too when you get tunnel vision (It must be asthma/It must be the DNS resolver... etc). Nice video!

ICS seems like a natural fit for so many professions. I wonder if there is a future for a Fire Officer to teach/adapt ICS for <insert profession>?

Volunteer Firefighter and SRE here.

ICS is crucial in any and all of our incidents and should be the model on how any disaster is handled

"While I go work with Slack for a couple years..."

Can't be encouraging for Slack, couple of years. But, that's the deal now, people leave tech jobs every 1-5 years for something better or start their own company.