> if your shit breaks, you fix it
What if you're drunk?
What if you're drunk?
I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers).
You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc.
Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it.
In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).
I am always on call to secure my own house, so I have a good alarm system, cameras, locks etc which means I hopefully don't have to do much. It doesn't keep me up at night when I'm on holiday because, well, it's overwhelmingly likely that nothing will happen.
If I were on call for the neighbourhood or general area once a week, I'd probably have to physically be there patrolling it because otherwise I have liability for things which are outside of my control.
It's not as if this is some strict 1:1 relationship, we obviously had more than 1 person with knowledge of / working on a codebase.
It would have been an issue if 2-3 working on the same thing left simultaneously, but for more reasons than just on-call.
I also gravitate towards teams that work well without a lot of documentation, so that helps in case I'm really unavailable.
On the other hand, if 24/7/365 is more like a 'nice to have' option, then sure, such an arrangement will be more cost effective than more people in support; a 99% uptime SLA often can be maintained by one core person.
A competent person can dig around and fix most anything good enough in three days and that leaves you 12+ hours for other outages.
Three nines is almost 9 hours a year, but planes almost universally have wifi now, even if it's not great. If you have a bad year, with two big incidents when you're on a plane, you might not make your goal.
At four nines, it's about an hour. Good luck with that one. You're kind of screwed either way. Someone with intimitate knowledge of the system can probably handle an incident or three in time, but it's an enormous amount of work to get people up to speed for that and you can't have very many live training sessions because you don't have the outage budget. At the same time, if you page me and I'm at the cinema, I'm not going to fix it in time.
But, the biggest thing is working to make failure as graceful as possible. As much as possible, each component should be able to fail individually, without affecting the rest of the system. And, when there is an outage of a critical dependency, you need a way to push that outage status upstream and a method to manage load when it comes back up. (Those knobs should be stable and can certainly be documented for others).
Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again.
In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!
To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better.
Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).