A practical guide to incident management
incident.io
incident.io
A lot of things feel unintuitive with security incidents because you are not solving a technical problem but addressing compromised risk and malicious human intent. For example, it is ok depending on the scenario to both cause an outage for seemingly trivial reasons or let a hacker roam free as you monitor their progress. KPIs by management that move from IT to security also have similar issues like measuring how long it takes to eradicate a threat as if it was an IT outage when you should take as much time as you feel to eradicate a threat (but ideally, but not always, the threat should be contained) because understanding scope of a compromise or attacker intent is more important. While more important KPI like dwell time are ignored.
I work at incident.io, where we’re building an incident management platform that helps your organisation deal with incidents at scale.
Today, we’re launching a set of resources intended to help those building their own incident processes that we’re calling the practical guide to incident management.
It includes chapters on:
- On-call, what it means, who’s involved, and how it should be compensated
- Foundations, defining an incident, and advising on how you measure them
- Response, for how to nail actually responding to the incident
- Learning & Improving, with pragmatic tips on how you can level up and learn from your past incidents
If you want to adopt pragmatic, battle-tested incident processes, or are looking to fine-tune an existing process, this guide is for you.
For those familiar with the content, this guide differs from the Atlassian and PagerDuty response content in that it:
- Acknowledges the way companies are working today: distributed teams, reliant on tools like Slack.
- Focuses on process and practice outside of just responding to an incident
- Provides actionable advice: real steps you can take right now
- Extends beyond engineering, reflecting our belief that incidents are an entire company affair
Very complementary content, but with different aims.
We hope you find it useful!
I take your point though, and if being on-call is having any meaningful impact on the time you spend working outside of hours, I agree $350 is low. For us, we're an early-stage startup with low numbers of alerts (long may it stay that way!) and the impact is low.
Not to try and defend it too much, but it's worth pointing out that the $350 is there to cover the inconvenience of being 30 mins from a laptop, rather than compensating for the time. We also give folks time off in lieu for any time they spend working outside of hours.
I would never accept on-call work (even if paid), even if no incidents ever happened because of the infringement that has on your lifestyle. Realistically it means I won't be able to attend church the Sundays I'm on, go for a long run, on a hike (if we have one of the only nice Saturdays of the year), or even visit family (some of my family have a terrible internet connection!), because even if nothing happens then there's the risk something _might_ happen: and if it did, I wouldn't be able to respond if I were doing one of those activities. And for me, no amount of money would be enough to compensate me for that.
My answer is yes, the on-call compensation is more about keeping healthy dynamics of on-call than it is about materially impacting your total comp.
I've worked in places where on-call pay was material before, and while it's nice, it comes with disadvantages too. As an example, you try hard to keep shifts balanced because if someone covers you and you don't do a shift that month, you notice it in your paycheck.
That's not always a good thing, if you want to encourage people to switch shifts to prioritise their health and reliable on-call cover.
There is something I've never known the answer to, perhaps someone here who knows more about incidents can help.
I have joined a small team. There are perhaps 2-3 devs who can actually fix issues. There are lots of issues, e.g. one incident a week. We're obviously trying to fix the software but every time something goes wrong it's a new problem. We have 100+ microservices interacting and 1k req/second, on a 20-year old codebase where most of the services are written by people who are no longer at the company, so although we're trying and making improvements, and I'm doing my best to help, it's not going to happen overnight.
Which brings us back to incident support. We're too small a team to have an on-call rota (i.e. with 3 people everyone would have to be on call once every 3 nights, apart from when people are on vacation or sick when the remaining people would have to be on-call more). So the management's idea is just that everyone have slack on their phone and whoever is available jumps in either when an alert goes off, or when someone (e.g. support) writes something isn't working. "Professional jobs aren't just 9-5, they do involve overtime sometimes". To be this just feels like being permanently on-call, which I don't want (burnout risk etc.). It might be different if there was an incident once a year but it's much more frequent than that.
So I know what I think we shouldn't do, but what should I advise the company that we should do?
We can't fix all the issues overnight, every incident seems different, we have too few people to have a rota, the software is too complex to outside operations to an external team (e.g. last fix last night was for one of my colleagues to deploy a quick hack to the codebase to turn off one feature, then build the code and deploy it), yet it's a international consumer website that should be available 24/7.
2. Read Google’s SRE book for ideas. For example, read about error budgets in this chapter: https://sre.google/sre-book/embracing-risk/
If the goal is to have a reliable system, then the ultimate creators of unreliable systems must be made to feel the pain themselves.
For example, product can be made to “feel the pain” by not being allowed to have any new features while an engineering team fixes user-facing or business-critical bugs ultimately caused by an endless plethora of new features being requested by product on short timescales.
In one word: backpressure.
Of course, this relies on product and/or management buying into the idea, which is why data can help you argue your case. The rest of your organisation might be more receptive if you can show them how much time is being spent on incidents and bug whack-a-mole.
The SRE book is an OK idea but to be honest, it's hard to get developers to focus on Google-level theory of reliable engineering via error budgets when they're so operationally immature that nobody is oncall (so everybody is oncall). Sounds like the gap is too wide. If these alerts are 50% "customer reported an incident" instead of a SLI, then that SRE book could spawn a cargo cult that creates work for everybody and achieves nothing because the gap is too wide. I've seen clueless managers latch onto theory that's too far ahead of the stone-age. Like "Hey guys, this month I want everybody to record their SLOs", when nobody is even recording metrics because they don't know how, or what to record.
I'd do extremely boring stuff like keep a spreadsheet of incidents with columns for severity, trigger (eg release, operator error, customer load, hidden bug, system error), root cause (eg internal software bug, bad config, system resources etc). If patterns emerge, they should point to gaps in knowledge, testing, automation, monitoring etc. If there's no pattern at all, then your engineering org is in a bad way. When an engineering department is extremely dysfunctional, it needs external help. Or sometimes it needs a clueless manager / lead to quit, allowing suppressed talent too take over and fix stuff.
Primary duty of on call is triage, pushing all non-critical issues to work hours and only bringing issues they deal with through mitigation and not resolution.
Assuming the latter, my suggestion would be:
1) Setup a weekly on-call rotation, where one of the 3 people who can fix issues is dedicated on-call (this means the other 2 can properly switch off) 2) Find some other folks who might not be able to support immediately, but are happy to learn. 3) Configure a shadow rotation so there's always a primary responder who know what they're doing and a shadow responder who can learn on the job. 4) Make it clear the shadow person has the same expectations in terms of turning up, but their job is to ask questions, write things down, etc.
I did this in a former job (and wrote about it here https://monzo.com/blog/2018/09/20/on-call) and it worked pretty well. After a few months we had a healthy rotation of 8 on-callers, and a waitlist of shadow folks who wanted to join too.
This advice obviously came from a point of near-zero context, so if you'd like to chat more I'd be happy to!
Is this a 24x7 thing? If so, this is the worst idea ever. I would quit immediately. Screw that manager. Honestly. This is clueless, and it shows a lack of care for the whole team.
One feature I was hoping this would have, related to on-call, but for minor week-to-week incidents:
My team gets many low priority alerts every week that need to be triaged, worked on, or ignored (something on the scale of 10 - 20 alerts per week, often re-occurring, often firing with slightly different tags).
These alerts are "symptoms" of something wrong with the system, and the oncall always has to manually sort them into "causes". A "cause" is then tracked and worked on, with investigation notes etc.
I've seen this workflow at a number of companies, and it is always handled with poor automation or none. Oncall engineers manually copy-paste the pagerduty - alert url into a spreadsheet (or google doc), manually assign a status tag, manually group re-occurring alerts into the same row, manually assign engineers (and the assigned engineers have to manually ack being assigned!).
Most of the time, the "symptom-grouping" decisions have to be made by a human, since a lot of the time it requires investigation to determine the link between two symptoms. But just because that one decision has to be made by a human, doesn't mean that all the bookkeeping shouldn't be automated!
With all these manual steps, there are so many places for human error.
We would pay so much money for a tool that did this for us
What also needs to be said - in my time I have seen orthodox ITIL bros, who got all their certificates and wanted to transform every ITSM process to the letter of ITIL and these people would probably get a panic attack from this guide. Nowadays I am seeing Agile bros, who want to do everything the Agile way, replacing one mess with another, just for the sake to be up to date with the newest frameworks.
I like that people at incident.io tailored the Incident Management to their needs and are not sticking to some predefined "rules" which are popular now. What works for them might of course not work for the rest.
I see the guide less as a set of dogmatic rules to follow, and more of: a) a set of sensible defaults for small to medium size orgs who are starting with little-to-no process and b) a source of inspiration for larger folks who maybe want to bring their processes out of the ITIL ages and into a world that's a little more applicable to the way folks work today.
As you say, the extremes of the ITIL <---> Agile spectrum is likely to be a mess, and where you should target on that spectrum is highly dependent on your starting point, your culture, and your appetite for change :)
This guide is how we ran incidents, or at least at the top end of the complexity of each section.
So if by small you mean companies under 1500, then I'd agree. But also, that's a lot of companies, in fact the majority of them!
We definitely see success with our customers up to the 1500 range adopting these practices, and often throwing out more convoluted or obscure processes that came before.
FEMA has something called the National Incident Management System, which is used when large scale bad stuff happens.[1] Many municipal employees get training on this, so they know what to do in a crisis. Incident commanders require substantial training. There's a certification system and even wallet cards, to keep clueless officials from trying to run things. It's not that complicated, but it's been thought through by people with dozens of disasters behind them.
There's always a single incident commander. It's usually somebody like a fire chief, not a political official.
A good example is how Walmart was among the first to bring water during Katrina:
> On-call payment is not expected to cover any time you spend working outside of hours. If you're paged and end up working in your evening, you should take time off in lieu. We trust you to manage this time yourself.
I'll make sure this is added this to the guide. Thanks for the nudge!
[0]: https://incident.io/guide/on-call/compassionate-on-call/
Is this an American thing? 5pm is not “nice and early on a Friday”! 4pm is about what time you should be finishing, and 3pm is “nice and early”.
Love, Australia x
My old job where I was last on call did Tuesday - Tuesday with a 1pm handover. It gave you time to deal with any issues that were still hanging around from the weekend but then you were done. Handover was happily during the work day so it could be moved back or forth a bit if needed and the new person on call had a couple hours to ask extra questions if needed.
Happy to say - having opened an incident - that we're now back up, and don't expect to hit any more issues.
Please try again, I promise it's worth another look!