I've worked as an oncall for a fundamental backbone service of the internet in the past and paged into middle of the night outages. It's harrowing and exhausting. Cool names like 'incident commander' do not change this.
We also had a "see ya in the morning" culture. Instead I'd be much more impressed to have a "see ya in the afternoon, get some sleep" culture.
I've led teams for over a decade that have oncall duties. One principle we have lived by is that if you are paged outside of hours, you take off the time you need to not hold a grudge against the team/company. Some people don't need it, some people take a day off, some people sleep in, some people cash in on their next vacation. To each their own according to their needs. Seems to work well.
We also swap out oncall in real time if, say, someone gets paged a couple nights in a row.
If you have two hours, you can take the training that describes the nomenclature behind the Incident Command System, and why it became a thing:
https://training.fema.gov/is/courseoverview.aspx?code=is-100...
This online training takes about 2 hours and is open to the general public. I took it on a Saturday afternoon some years ago and it gave me useful context to why certain things are standardized.
Here's all the content on one page, I found it less intimidating than the 50 slide format https://emilms.fema.gov/is_0100c/groups/133.html
ICS, at its core, is a system for helping people self-organize into an effective organization in the face of quickly changing circumstances and emergent problems.
Some simple rules are things like:
* The most senior/qualified person on-site is generally in charge. (How you determine that kinda varies depending on organization.)
* Positions are only created when required. You don't assign people roles unless there's a need for that role.
* Positions are split and responsibilities delegated as the span of control increases beyond a set point.
* Control should stay as local to the problem as it realistically can while still solving the problem.
From there, it goes on to standardized a template hierarchy and defines things like specific colours associated with specific roles so as roles change and chaos ensues, people can continue to operate effectively and in an organized manner. In-person, this means things like the commander/executive roles running around in red vests with their role on the back. If the role changes hands, so does the vest.
Some of the roles in that template organization are things like:
* The "Public Information Officer" who is responsible for preparing and communicating to the public. This makes a single person responsible to ensure conflicting or confusing messaging is not making its way out.
* A "Liason Officer" who is responsible for coordinating with other organizations. This provides another central point of coordination for requests flowing outside of your response.
I think we could all imagine how this starts to become valuable in, say, a building collapse scenario with police, fire, EMS, the gas company, search and rescue, emergency social services, etc all on scene.
In an IT context, what this means it that, generally, the most senior person online is going to be in charge of receiving reports from people and directing them. If there aren't many people around, they'd generally be pitching in to help as well.
As more people show up and the communication and coordination overhead increases, they step out of doing any specific technical work. If enough show up, they may then delegate people out as leaders of specific teams tasked with specific goals (they may also just tell them they're not needed and send them to wait on standby).
All roles, including the "Public Information" and "Liason" roles fall to the Incident Commander unless delegated out. At some point, if the requests for reporting from management start interfering with their role as Incident Commander, they delegate that role out. If it turns out the incident is going to require heavy communication or coordination with a vendor, they may delegate out the Liason role to someone else.
ICS is probably largely unnecessary if your response never spans larger than the number of people that can effectively communicate in a google meet call, but as you get more and more people involved it contains a lot of valuable lessons and things learned through real world experience in situations much more stressful and dangerous than we ever face that help you effectively manage and coordinate the human resources in response to an incident.
(Disclaimer: That's all basically from memory. The city sent me on a ICS, ICS in an emergency operations centre context, and a few more courses a few years back as part of volunteering with an emergency communications group. It's probably 90% accurate.)
The Incident Command System contains guidelines to help different organizations work together in an emergency. You might need the fire service, or police, or hazmat, or EMS, possibly across multiple jurisdictions. If I recall correctly, ICS bans all those "10-4" radio codes, standardizes job roles, etc. And it contains rules for scaling the temporary organization up and transfering leadership as necessary.
Overall, it seems well-suited to real-world emergency response. And the training materials recommended using it in non-emergency situations, too, such as large parades. The idea was that major emergencies are rare, but it's worth getting practice with more common events.
I'm not quite sure how well the ideas behind ICS applies to IT outages. Most outages occur within a single organization, so there's less need to coordinate with outsiders. But some of the advice for scaling the response team and handling leadership transfers might be very useful.
In that sense I could still see some value around the ideas of organizing based on whoever actually shows up when the 3AM call goes out, adjusting roles on the fly as the situation evolves, and assigning out specific people responsible for updating management, interfacing with vendors, etc. And I think it puts people in the right mindset to do all that.
But yeah, I wouldn't like... plan on sending all my staff out for ICS training next week or anything. I think there'd be some value in sending a couple people responsible for creating and implementing your disaster planning to take it though--there are a lot of good things you could copy, some you could modify, and some you could throw away to make a pretty damn good IT-oriented response procedure.
And yeah, as far as I've seen our city actually uses ICS any time the emergency operations centre is activated. That's whenever there's a large-scale emergency that requires more centralized coordination, but also sometimes just if there's a large event that requires coordination (e.g., a festival with 100k people showing up at the beach). It's not just for practice, but because it's just a system that everyone's already familiar with that can respond to the situation if it evolves. Nobody needs to make the call that "we're switching into emergency mode now" and stop everything to reorganize, they simply respond.
German labor laws forbid employees from working 10-13 hours after a long on-call situation after a normal work day, just like that. Add in time compensation, and a bad on-call situation at night easily ends up as the next day off paid.
I've found this to take a lot of edge of on-call. Sure, it /sucks/ to get called at 1am and do stuff until 3, but that's a day to sleep in and recover. Maybe hop on a call if the team needs input, but that's optional.
I think that this kind of drill helps a lot for cases where you can take a pre-planned route, like deploying that backup server or rerouting traffic. But the obvious question then is: Why not automate that as well?
When it comes to diagnosis or, worse, triage, in my experience you want independent free agents looking at the system all at once. You don't want a warroom-like atmosphere with a single screen but rather n+1 hackers focusing on what their first intuition tells them is the root cause. In a second step you want these hackers to convene and discuss their root cause hypotheses. If necessary, you want them to run experiemnts to confirm these hypotheses. And then you decide the appropriate reaction.
It’s probably nothing but overestimation but I feel like I’m seeing more of this later in my career than I did early on, or maybe I’m paying more attention?
Whatever it is: past experience (which includes coming from a military family in the states) has taught me to avoid companies that crib unnecessary amounts of jargon, lingo and colloquialisms from the military.
Curious if others have noticed or even feel the same and what your experiences have been for feeling similarly?
I support the notion there's a strong difference between lingo that's properly applied to the situation and lingo that is recklessly applied because it "sounds cool".
The examples you gave seem to be fair game for the work being done-in the interest of brief, specific language; the examples I gave in another comment though ("flanking","breaching") however are just grating and...weird to use in a work environment.
Your point is nevertheless well met.
"dvtrn you are to take the flank and breach this issue with Paul"
And this ran all the way up to the top of the org. Senior leaders were constantly quoting that Jocko Wilink fella. It was...something.
My old man (a former Dill Instructor, made for an interesting childhood) found it utterly hilarious when I'd call him up randomly with the latest phrase of the day, uttered by some director or another. To my knowledge, and I sure-damn asked, the only affinity anyone on the executive team had with the military was two of them having buddies who served.
Like Agile, ICS is a set of principles that work well if you are properly trained in the system. Unlike Agile, ICS has not yet been buzzword-harvested by clueless managers to get promoted, so the concept itself still has some value.