I get that the point of the article is to sell the value of run books, which are amazing, but are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Even with perfect run books that seems unwise.
But I work in e-commerce, not life-critical systems of any sort..
The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch.
As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people.
I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years.
As such the first entry in my run book is “judging importance”.
Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that?
And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record.
On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.
I still get calls about some service or application I have never heard of.
Technical debt is real and I lose a little bit of my life every on-call week.
And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure.
Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.
That sort of thing is literally the only thing I miss about the office.
Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
Live root-cause analysis of a failure which previously occured in Production - you explain your thought process (check connectivity between System A and B via ping, DNS name resolution, look at open ports, etc...) and they tell u the result like "ping returned 'Name or service not known'".
SWE role in infra.
Yes. Very common
> Even with perfect run books that seems unwise.
Yes
But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.
Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.
No amount of training is enough as features and code gets deployed every day.
It was horrible and I left.