Diary of a first-time on-call engineer
thenewstack.io
thenewstack.io
If an enterprise thinks their applications should be up 24/7 they should staff for 24/7 support.
On-call is a way for bad businesses to steal money from their workers by expecting them to work 24/7 without paying them 4.2x their salary.
And even if they did pay a full wage per hour on call it wouldn’t be worth it because nothing truly compensates you for the hours of your life you’re losing while tethered to your work phone and computer.
Let's go over that again: People liked being on pager duty, and all it took was a few hundred bucks a week.
Of course, this was in a pretty competent company. The pager would only go off a couple times a week, it was usually easy to fix, and there was a well-documented escalation path if you couldn't handle it on your own. So we weren't losing sleep.
If the pager started being painful, fewer people would sign up, giving a signal to management that there was a problem. They could then either increase the bonus, or fix the root problems.
I have no idea why other places don't handle it this way. It worked well for everyone involved.
That's why you earn so much more during on-call in Germany
>Bereitschaftsdienst Beschäftigte im Bereitschaftsdienst halten sich in der Regel im Unternehmen oder in unmittelbarer Nähe auf, damit sie die Arbeit bei Bedarf sofort bzw. zumindest zeitnah aufnehmen können. Ein typisches Beispiel sind Feuerwehrangestellte, die auf der Wache die Nacht verbringen, aber dort schlafen können. Bereitschaftsdienst gilt in vollem Umfang als Arbeitszeit und muss daher bei der täglichen und wöchentlichen Arbeitszeit voll berücksichtigt werden. Die Vergütung des Bereitschaftsdienstes richtet sich nach dem jeweiligen Arbeitsvertrag, beziehungsweise gültigem Tarifvertrag oder Betriebs/Dienstvereinbarung.
>Arbeitsbereitschaft Bei der Arbeitsbereitschaft sind die betreffenden Beschäftigten im Zustand „wacher Achtsamkeit im Zustande der Entspannung“ am Arbeitsplatz anwesend. Sie sind jederzeit einsatzbereit. Ein Beispiel könnte eine Beschäftigte in einem Elektrizitätswerk sein, die beim Piepsen eines Überwachungsmonitors sofort aktiv werden muss, über längere Phasen aber keine Aufgaben zu erledigen hat. Arbeitsbereitschaft ist Arbeitszeit und muss im vollen Umfang auf die tägliche und wöchentliche Arbeitszeit angerechnet werden.
>Die Vergütung richtet sich auch hier nach dem jeweiligen Arbeitsvertrag, beziehungsweise gültigem Tarifvertrag oder Betriebs/Dienstvereinbarung.
To add to this: If your boss calls you when you are on vacation, you get that full vacation day back. If your boss asks you to be ready in case he might call during your vacation, you also get those vacation days back.
So… not being able to: go on a long run without the phone, go watch a movie, have a big nap, go on a hike without a laptop on you.
Doesn’t sound free to me. My last on call wasn’t bad. But I still didn’t like being bound to my laptop and phone.
1. There is a rotation.
2. There are few pages, preferably the median should be 0 per week.
3. Spurious / non-actionable alerts get fixed right away (with very high priority)
4. You're not up more than 1 week per 1-1.5 month.
5. You subtract middle of the night pages from your next working day, with bad nights resulting in a day off. Being on-call doesn't mean working overtime.
Nobody should be coerced into on-call within a role that did not explicitly require it on joining.
Rolling it into the contract isn't appropriate because anything that is "free" will get abused, eventually. Even if it's fine right now, it won't be eventually. And there's no way to stipulate that the "median should be 0 per week" in the contract, so that'll go out the window as soon as the company needs it to.
I upvoted you because it's obvious you've thought about a lot of the problems with this and I feel you're on the right track. You just didn't go far enough.
This is not entirely true. I think the fact that companies don't have to hire separate support/SRE staff is priced into the exorbitant salaries that devs are paid these days. When I started my career around a decade back, there used to be at least 2-3 levels of escalation engineers/support staff between the customer and the developer. Today it's usually zero, and part of those salaries paid to dev instead.
I may be in the minority, but frankly, I'll easily take a pay cut not to have to go through the on-call sh*t every few weeks.
Is 54k for 60 hours per week considered exorbitant 10 years ago? Because that’s how much I was paid while being expected to be available 24/7. My cheap apartment was a 3rd of my pay. I was fortunate to have no debt and get a car from my parents.
I was up to 90k about five years later. Which is decent in the Midwest but I wouldn’t consider it exorbitant.
I doubt I’ll see anything near $200k for a long time. By then, it’ll be less than 90k considering inflation.
My manager once realized that I was sometimes even called by the other oncall engineers if they were stumped, and retroactively compensated me for a "2nd level oncall".
Not even once have I ever felt exploited - it was always sufficiently flexible for my personal life, usually 1 week/month, and always reasonably compensated.
I believe the SRE book mentions this; if you want your software to go from on-call engineers to being managed by SRE's, your software needs to be stable, have certain logging requirements, documentation, etc. And, if it breaks more than X times in a row, or causes an SRE to be paged too many times, the pager will be handed back - fix your shit.
They were flabbergasted when I declined, and tried for weeks to get me to accept with "It will be good for your career", etc.
That is a VERY hard no from me.
The idea there is no middle ground, with
Want to meet with somebody: take your laptop with you in a backpack. Just make sure you have mobile connectivity wherever you're going.
Want to party: just ask somebody to take over the pager for the evening.
If you have <=1 page per rotation, then it's really not annoying.
The rotation is like a week every 2 months. You get paid a substantial amount extra for that week, and you get paid on top of that if you get a call out. Even if it takes you 10 minutes to resolve or there's no issue, you get paid a minimum of a few hours at 2x pay.
It's insane to me that there's places with multiple call outs a day and people don't get paid extra.
For me, being on call is pay rise for the week. It just means I can't venture too far away from home without my laptop. It's so rare to get called out.
I'm very well compensated for that, and have historically been more than happy to fire the contract "support" teams in India or wherever that are trained to follow runbooks to manually fix problems that should self-heal if systems were engineered correctly.
On-call is just a way for me, an overpaid engineer who cares about craftsmanship, to ensure that problems serious enough to warrant a page (infrequent) are dealt with swiftly, and followed up on with an in depth post mortem to ensure they don't happen again.
To be clear I consider getting paged more than two or three times a year unacceptable. Any compamy with on-call should track life interruptions as a KPI and actively work to drive their frequency to zero.
I build high scale systems for a living, but I also hike, ski, and have a family. I can't guarantee 20 minute response times 24/7 365 days per year. I generally won't join a team with an on-call more frequent than once per 6 weeks.
Anecdotally, companies fall into a trap where individual engineer's own services that they must be paged for. These engineers don't get the time to make the service reliable, and they also don't want to admit that the service is a tire fire. This works for the company as long as they can continuously pull in new engineers to feed the on-call fire.
I think that’s where I’d draw the line if I was in such a situation. If I’m on the hook for dealing with any outages, then I won’t be working on anything else until the service is reliable.
People need breaks.
It can, but I don't think it's too hard to make it robust enough that pages become a rare occurrence. At my company (a small startup with <5 developers - so we're not exactly flush with development capacity) we've managed to reduce outages in a previously quite buggy app to almost nothing over about 18 months (while also developing new features). I don't think we've had an outage in the last year which wasn't due to either:
- An outage at our underling hosting provider (which we can't do anything about anyway)
- A code deploy (which we can plan and control the timing of)
But then she goes into the "example week" including a weekend page. It seems like there's not much going on, not many pages, not much out of regular hours. And then this prime example of why you do not want to have engineers on-call:
At 5 a.m., I got paged. I jumped out of bed and ran over to my computer.
The page self-resolved at 5:01 a.m.
I get it, all the talk about "you will build better services because otherwise you will be paged". In the end I think it's just about saving money for dedicated 24/7 network operations if that is what your business requires. If it doesn't have customers 24/7, then don't set up any pagers and support staff, if it does, pay for it!
I've been on call in a SaaS environment before. In my entire time (counted in years) doing that I have had about the same amount of pages in total (!) as are contained in her example week. I carried my laptop and the phone to tether dutifully but I have never been paged by the 24/7 support staff for something like the 5:01 self-resolving alert.
Even if that "on call" was only 9-5, getting distracted and losing half a day to tracking down the latest flaky test or owner for a failing dependency really adds up and makes it significantly more difficult to do my day job. When I'm oncall, I'm oncall 24/7 for a week (30 minute SLO) and the expectation is that the amount of non-oncall tasks I'm able to get done that week is significantly impacted. I'm fortunate to work on a team that acknowledges that reality; I know many people, even at large software companies who should know better, whose management is far less reasonable.
Pay more. It literally solves all recruitment problems.
I joined a US company as part of a follow the sun model. Although ironically I seem to support people who are up at midnight in Europe more than in my area of the world.
Happy to have a chat, see if I can offer any useful feedback :)
Ideally I'd be paid extra as well (I get about an hours pay a week as an on call allowance). What makes it work is that I'm the senior dev on the project that I'm on call for - if it hits the fan it really is my problem. And more accurately, in the last two years I've been called out twice. So that extra hour a week of pay... it's just free money.
No. The downside of being on-call is not the number of times you get paged at 3am. The real downside is that if you are on call 24/7 and you have to react to an alert within the next 15min that means:
- you cannot go out for dinner
- you cannot go out for a run
- you cannot go and pick up your kids (if the ride takes more than 15 min)
- etc.
The only thing you can do is stay at home with your laptop next to you, or wherever you go, take your laptop with you. That's being on call, and it's NOT free money.
We don't have anyone on call in the sense you mean, and I've never seen it in the wild. If they want someone right there all the time they've paid them to be in the office where they have proper access to critical systems. When I worked in a medical environment that was what we did, and those people got paid very well for doing overnight "tech support" ... they were bored enough that answering random IT questions seemed worth while for them.
A lot of stuff can be resolved from your phone though.
After reading a bunch of the other messages here, I think the reason it was ok for me was that I was always in a position to fix the root cause. I wouldn’t ever get the same alert twice if I did my job correctly.
Running is an issue, unless you don't mind a sweaty back to carry your laptop, or just run loops nearish to your house with your phone on you.
What I usually did when picking up kids (close by, about 30 min round trip, YMMV) was to ack the page from my phone and log in when I got back home.
Yes it sucks to be tethered to a laptop. But if after-hours pages are rare, and you aren't on call more than ~25% of the time, for me personally it's acceptable
Of course we also had a night shift guy who would actually call you if he got stuck, so hell if I knew why the SMS system existed. (That role lasted about a year at most - night shift for tech is stupid).
(general observations over a longish career)
Especially for their helicopter, rover at least stays on the ground throughout all this.
And I developed this sort of 'anxiety' every time my phone rang. My stock ringtone never felt harsh before, but the instant my phone started ringing, I could feel my heart pounding. Would this be the call that would finally be more than I could handle? Even after I was done with my rotation, I just got so agitated any time my phone rang.
I switched my ringtone to the Mr. Roger's Neighborhood theme, and I left that team (for other reasons). I'm a much happier person at work and at home now.
Edit: I'm not in my 20s anymore and I have a family.
When there's an ops team responsible for on-call nobody gives a shit about fixing the code issues that wake them up.
"You build it, you fix it" has to apply to the powers that be (i.e. the product managers and such), not just the individual contributors.
While many organisations pay lip service to uptime and reliability, the reality is that those are not sexy or exciting and Bob in sales just snared this million dollar client that will sign as soon as we implement frobnication with Othr. So dev team can either work on fixing the reliabilty issue that means they don't get paged at 3am or implement the thing that get the company another $1m. Which do you think they get to do?
I had a very terse conversation about this, and directly included the words “lip service” to my superiors today about the very topic of on-call and overnight work this morning. I’m fed up, and made it known.
Because IMO you’re 100% right, too many orgs pay lip service to availability and SLAs, yet deliberately choose to underinvest in tooling and resources to help mature the capabilities of incident responders.
Does it bring it revenue? Probably not, but bailing water doesn’t move a ship across open seas either, you still do it because if you don’t surprise, no more fucking ship.
Instead of a well defined monitoring system I get $40/mo for the lowest tier status board that’s generated more false-positives than actual on cable alerts, no matter how I tune it or try to define parameters. Instead of a mature incident response platform I’m demanded to conduct Spreadsheet postmortems. Instead of centralized logging and telemetry I get to troubleshoot exceptions over a bunch of shell windows.
Incident response is still a mess of people slapping keyboards, there’s no triage, there’s no sense of structure or command of the process.
And here I am, an SRE being prioritized to work on backing up service accounts because we might not get a contract with this trucking company if we don’t have that capacity.
Sigh.
Unless you have a culture that values these things from the top down, you're going to give yourself a concussion banging your head on a brick wall.
(Joking, it’s really just a plain old sinus headache)
If you start from the assumption that your employees are not motivated to do a good job and so they need to be punished for doing a bad one… I’m not sure what to tell you.
It's not like devs choose what to work on & prefer the 70% solution before moving onto the next demand..
In every job I’ve been in, I’ve been responsible for bad code I didn’t build and had no authority to fix it.
This only gets worse when one team member can’t and never will be on-call. Because everything is focused on “individual responsibility” anyone who isn’t “carrying their weight” gets singled out, instead of actually fixing the systematic problems.
It’s much easier to blame people and make examples out of them than to actually address the real issues.
On-call rotations are the worst.
1. If you are also supposed to be available regular business hours, management treats it as an extra that according to them shouldn't take up a significant amount of time/energy. Therefore, you get compensated for it accordingly: a pat on the head and a fractional amount of comp time that you frequently can't use when needed anyway because you still need to close on your normal tasks.
2. Since it doesn't officially take up any time or energy, there is no problem expecting you at work on the dot when you have been on call putting out fires the night before.
3. Getting paged for nonsense is a real problem. If you were asleep, it can take you 10-15 minutes to figure out it was nonsense and another hour to get back to sleep.
4. Following on the last point, more than two or three pages during sleeping hours means you effectively pulled an all-nighter. Pretty crappy when you aren't 20.
5. When everyone is burnt out from the on-call rotation, it is difficult to close major issues, let alone the small things that generate extra pages.
1. I'd give you 2x the amount of time off immediately, the next morning. Minimum 2 hours. More than that, I'd ask if you'd prefer to shift your workday later or find some other flexible arrangement.
I have fought management at some places really hard when they said a word about someone's planned work bucket not getting done after being paged as much as OP has in a week. I've made it clear it was management's fault, not yours.
2. Absolutely not. You'd be given the option to get rotated off Pagerduty as soon as your life was disrupted. I'd personally take over your shifts if I couldn't find someone on our team who was willing to take over.
3. This is SRE priority #1 and I felt terrible for OP. We cannot run a marathon if we're always sprinting out of bed for crap. Alert Fatigue is real https://www.pagerduty.com/blog/lets-talk-about-alert-fatigue...
4. Bingo. So, the whole organization has to reduce velocity so we can pay back the debts. If the CTO doesn't totally empathize with this, they're going to trip over their own shoes.
I make better money than most people I know around my age, but it's nowhere near tech money and it's not enough for the bullshit. Unfortunately, it is good enough that pivoting into another field and taking a pay cut would be painful.
My sleep is wrecked for more than a week after a bad on-call - what I actually need is a week or two off to get my life back on track
I totally get the sleep thing, though!
Boy would I be annoyed if I got paged because someone else's commit made a test fail. This should be dealt with by the dev that pushed that change. The fact that it made a test fail should also mean it's not in prod yet, making me wonder why someone got paged in the first place.
I get that some systems need to have very high uptime, but it seems like we've become far too obsessed with the idea for a bunch of trivial crap.
Mainly because I expect the 1 hr-a-day shutdown would be in my prime time.
StackOverflow seems to go down a couple times a month, USPS had a few hour maintenance window, many banks still have huge maintenance windows.
Scale expectations.
Recognise that each additional point of reliability increases costs (and time to achieve) by roughly an order of magnitude.
This is in contrast to the model where there's a team that fields issues and tries their best to fix them, but ultimately has to put fix-it tickets to the software owners.
The on call rotations I've seen end up being a hybrid where you get the worst of both worlds. The team has enough responsibilities that no one person on the team knows how to handle everything and are so inundated with pages that it just becomes accepted that one person from the team every X weeks is going to be nothing but keeping the service from burning down.
The pager goes off and I return the page.
It was the trauma surgeon.
“Hi there I have a guy who shot himself in the head. There is brain oozing out of the hole in his head. What do you want to do?
Later on, I switched careers from neurosurgery to computer science.
My first time being on call for Google was not nearly as stressful.
My company has strict rules on what is allowed to escalate to on-call and the SLAs to handle that alert. It had to be breaking our overnight batch cycle or we’re not getting up. Brittle processes are stamped out immediately and management backs us up.
On using a public channel for coordinating incident response, we have struggled in the past with stakeholders joining the meeting and offering well intended but ultimately distracting input during the incident. We’ve found it best to have one responder play the role of “incident commander” and manage external communication / rope in more stakeholders as needed. This helps avoid conditions that make the incident even more stressful, like say a member of upper management demanding frequent updates or spreading FUD.
At the next job I was also on call for systems we put on a manufacturing line. That was a lot more exciting and involved driving to the plant and try to get things going when production stops. Losing $10000 per hour when the line is stopped wakes you up real fast even if you just jumped out of bed 20 minutes ago.
Another case of: can be done properly is a good work environment, and can be literal death when done badly.
This felt like a major red flag to me - why was the team that owned the offending service not the one receiving the page?
This is a trade-off in order to reduce the number of people who have to be on-call at a given time and maintain a voluntary model. Generally, though, the person on-call will be from a team that owns at least a portion of the services they’re on-call for, and the services are distributed among the two rotations thoughtfully.
I'm not sure if one is better than the other, but the assumption of only being on-call for services you own assumes some minimum team size, or as acceptance of being on-call every other week.
1. Learn about X.
1.1 See that awareness of X is distributed through the team.
2. Document that X is not documented. Have X dev / support team document X. Configuration, monitoring, troubleshooting, recovery, escallation.
3. Scale expectations. If X is not documented and awareness is limited, then issues involving X are not expected to be resolved by on-call staff and will be addressed when X dev team are available.
4. Scale opportunity/risk of X based on the actual experience and downtime it affords.
SRE is fundamentally a risk-management exercise. Manage the risks. Assess them, mitigate them, reduce them. And measure the damned costs of them.
Today, my job is very different. It's a whole different playbook. 6 months into my role, I was asked to go on the on-call rotation. I learned enough to get through it, thankfully one of my colleagues were available in case I hit a situation I couldn't handle. Almost 2 years into the job and I don't sweat it anymore, I know enough of my service to fix most issues and now appreciate the escalation procedure. I will kick it into gear, even if I need a storage admin to give me a few GB of disk space to get me through the weekend. :)
If that was me I'd either take myself off the on call until the false positives had been fixed or look for a new job if that wasn't allowed. I suspect that this was forced upon this person.
Being bubbly and nice about being abused like that is terrible and bad for the industry as it signals that you can get away with accidently calling people at all hours. I would say for each false positive a top priority ticket should have been put in such that all work is dropped until it's fixed. The "What I Learned" section is mind blowing. She thinks she got lucky !
Sounds like the company has many problems though like "a piece of system architecture due for retirement and no longer serving customer traffic" I mean wtf !
I still get triggered when I hear the Nokia ringtone.
First, we were all offered good rates for on-call support so we had plenty of people who wanted in so I wouldn't do any support and leave it to the guys who wanted the money.
Then some of the guys left the team and got replaced, they offered a different deal to the new guys, less lucrative, therefore, they didn't want to do it.
As a senior I had to start doing support, I thought the extra money would maybe make me feel ok, then we had less and less people on rota so I was doing support most weekends which started affecting my marriage.
I escalated the issue (again), nothing was done, one month later I gave my notice on the highest paying and best dev job I had.
I was on-call for full month straight during my notice, that's how bad it got. And I steer clear from dev with support roles now.
(edit: and by "hackathon" because I realize there's a bit of room for ambiguity, I mean internal corporate Hackathons where the company "allows" dev teams to take a "break" from "regular" work in order to ideate and build new features and functionality that--if their idea gets chosen--becomes "regular" work in and of itself. It's those kinds of hackathons I've started removing myself from)
>> We spent about 45 minutes looking at the alert catalog, the runbook for the services and trying to fix the underlying problem. After getting more information, we realized it was not affecting customers and could wait until the team that owned the service came online.
Well, now, that is not a sev 1 alert. If you're on-call, you have the power to adjust alerts! Change it to a sev 2 so it does not alert future on-calls unneccessarily.
Amen.
Sounds like the author went even further: rather than just tweaking an alert, she deleted one, then deleted the whole service. That won't page again!
> Change it to a sev 2 so it does not alert future on-calls unnecessarily.
Or restructure the alert. I prefer SLO-oriented pages like "we've burned 1% of our quarterly error budget in the last hour" rather than cause-based pages like "CPU usage has exceeded 90% for 10 minutes". If you are getting paged based on something that doesn't affect customers, it might be a sign that you have the wrong alerts so you can't tell if they represent something important or not. Manually toggling the severity each time it's wrong might not be good enough.
In terms of overall operational load though this week doesn't look bad at all: four surprise pages in seven days. After her fixes the next week should be lighter (knock on wood).
IMHO there are two angles to achieve higher reliability:
1. Build systems which go down less on their own (which is quite difficult)
2. Get good at fixing them fast when they go down (which is much easier)
A trivial example of 1 might be moving from no RAID to RAID1, an example of 2 might be getting on-call proficient at quickly responding to a dead disk and restoring from backups.
(But I did say 1. was hard, even in this simple example maybe you used hardware raid, and are finding out just how crappy HW raid can be ;)
Companies attempt both angle 1 & 2 to differing degrees. I worked a lot in 2, and it sucked. We were a proficient fire department in a city of straw houses and gas stoves.
Aside from the general suckage of on-call (24 hour days, many, many nights lost sleep), the work is by it's nature high risk, high urgency, high impact, high risk - but is rarely rewarded sufficiently in pay or promotion opportunities.
Having spent so much time on angle 2, I've decided it is largely shallow work, and to grow intellectually I needed harder more interesting problems, which meant trying angle 1.
Consequently now my title is SWE and I don't do out of hours on-call, and it is glorious. I do notice I see the world differently to my less scarred fellow SWE. It definitely changed how I write software, and view the product priorities.
At heart I still consider myself an SRE. It's just I had to place myself in the most impactful place to achieve it, as a regular product/feature dev.
I'm staying primarily for the money but I'd rather do something else and keep programming as a hobby.
Only one of these is a real problem, the other 2 are essentially derived from that.
First of all, I appreciated this diary because Anna took the task with a positive attitude and as a learning experience. Thanks for writing this. To see an old problem through new eyes is inspiring.
I have numerous, "hot-take" criticisms of your current organization's practices, but I'm not sure I have all the context yet. The one suggestion I will make is: if you're not already using it - clone https://github.com/pagerduty/incident-response-docs/ and modify it to meet your organization's needs. Then, have it blessed as policy by management and train SREs and Devs on it.
To the other comments: I see there's a lot of people here who say they'd never do the SRE job, or return to doing it. I'm not discounting your fear or feelings of burnout. Been there. But, hear me out:
DevOps is not just about CI/CD pipelines and monitoring and Pagerduty. It's about having a culture where developers don't throw operational or security poop over a wall of confusion at sysadmin types as well as at their peers. This kind of organizational dysfunction can be devastating to a business.
DevOps at it's best is about about empathy. One of the best places I ever worked was filled with developers who had true empathy. They realized that an error or omission in their work could would wake up their Ops team at stupid o'clock in the morning, repeatedly - leading to all the things that drive SRE's and on-call folks literally insane. They practised strict TDD.
These developers volunteered to be second-tier on call after the ops team did triage, out of the kindness of their hearts for their coworkers. Management also led a culture of defending time to find permanent solutions to drive measured improvements in SLI.
SRE isn't about waking up at stupid o'clock every night to press buttons. It's about having a culture of driving permanent fixes and compensating by using cost-effective and appropriate cloud architectures. It's also about leading the working agreements with engineering teams to do blameless post-incident retros together and making the work bring your teams closer instead of pushing them apart.
I can't help but take away that a lot of you feel like On-Call heroics are what SRE is about. It's more difficult than that, but also less stressful, and simultaneously more rewarding when you get it right.
You were very very lucky. Most of my on-call stuff has been constantly the same issues again and again. I have stopped caring. Server A for Customer B fails updating because customer B refuses to pay for extra disk space, time and time again. Corporate president C cannot email because the idiot does not respect maintenance windows he himself agreed upon. Refuses to pay for more server resources though...
I like to promote ownership by making each engineer responsible for their area. So, they must understand the business so that they can judge if a fix can wait.
We're fortunate to have a 24/7 customer support team who both attempts to identify quick workarounds and aggregates reports before notifying the head of the relevant team.
Firstly: when was the last time your on call rotation comprised exclusively programs that you had written? How many systems does your team look after that it "inherited" from elsewhere?
Secondly, how many organisations allow developers to prioritise solving pages over other forms of revenue generating work? How many teams demand negotiation with a "product owner" before a piece of work can be added to the board?
Thirdly, how many pages are truly easy or tractable to fix? I have worked in teams where we were constantly paged due to external API failures, but intra team disagreements meant we could neither punt the responsibility elsewhere nor resolve the problem. The problem wasn't technical, it was social, but there's no PagerDuty for absentee Engineering Managers.
Fourthly, how often is the Real Problem™ for the reliability and uptime of a given system, actually at the program level, and not at architecture or system level? How many programs are hamstrung by internals alone? Once you get past the basics, most of the _really_ significant decisions affecting reliability are at a system design level that Johnny or Jane Developer isn't going to be empowered to fix in a Scrum "sprint".
Really, this "shitty on call incentivises robust systems" argument is facile. It's paper thin. Put it out in the sunlight for a second and it crumbles. It only makes any sense, tentatively, under idealised conditions where developers alone are responsible for non functional requirements and the usual relationship of employer / employee is suspended.
Think about it for a second. It's just rot.
* You are only on-call for systems you can directly change.
* The person who is on call has full flexibility to work on anything that improves the life of the on-call person for that time.
* Anything that wakes people up in the middle of the night gets prioritized above new feature work.
* There is a pager set of monitors, and a message only set of monitors. If a page goes off, and there is nothing the on-call engineer can do about it, it gets moved to the message only channel or removed, because it is a bad monitor.
I discussed this list of rules when I interviewed and the job description included being on-call. It wasn't a negotiation. If those aren't followed, I remove the monitors. I'm sorry you had a terrible work environment, but I encourage everyone to have professional standards. I hope you find a place where you can.
Agree it's not fair to be on-call for things you can't fix. That's silly.
But, you're surely not saying when you push a change to a 24/7 mission critical system, and you're on the "git blame" - you're not interested in taking responsibility if the consequences occur after 5pm? And now an SRE has to learn everything you know, in order to fix it, instead of asking for your help?
Also - think about it a different way, you personally, are not the one I'd ask to participate in on-call. It's your whole team, maybe even your whole department. You and your team/department should work out how to field that together because you have more context into each others work/life balance, projects, skills, risky commits, etc. Your team would manage your own rotation.
> Secondly, how many organizations allow developers to prioritise solving pages over other forms of revenue generating work? How many teams demand negotiation with a "product owner" before a piece of work can be added to the board?
Tell me about it. Totally agree. This is where I focus much of my energy as a part of leadership teams - making other leadership realize they're over-promising on feature delivery without concern for their poor operations.
Frankly, fuck sales-driven development for this. I even work with marketing departments to value-stream map so we include good ops & security as part of our core prop to customers and investors. If they don't get it? Well, I guess the fish is already rotting from the head, as they say.
> Thirdly, how many pages are truly easy or tractable to fix?
If they're repeated ad nausea? Then those alerts need to go. Now. And if they can't go, because Management don't see the pain they're causing? See my response to #2. Culture change first.
> Fourthly, how often is the Real Problem™ for the reliability and uptime of a given system, actually at the program level, and not at architecture or system level?
Depends on your industry/platform. Good SRE sysadmin troubleshooter types will do everything they can before they push the escalation button to engineer tier. At least they'll get all the data and metrics they can and document the situation at hand before the engineer comes online.
Sometimes you don't really know what caused a problem until you all get together and retro it. Better to stay blameless even beyond when you have consensus on the technical factors.
One time, it was a leap-second messing up some 3rd party vendor software, with consequences for our code base that nobody ever expected.
Another time, it was a tired SRE combined with a lack of proper procedure combined with a code mistake with no test coverage.
Another time, it was a binder that fell on a customer's space bar effectively DoS'ing a system that had no rate limiting.
Some of those got fixed pretty quickly by an SRE.
BUT - whatever the cause... the times that our MTTR got truly trashed? (We're talking DAYS...) It was because the person who wrote the code and had all the context, had already quit long ago - leaving us with no proper logging and no one to help us. The organization had never prioritized replacing their code or hiring someone who could understand it.
> Really, this "shitty on call incentivises robust systems" argument is facile.
You're right in the sense that, considering all of the above, I would need your buy-in. To get it, I'd seek to eliminate the "rot". I'm truly sorry you got burned in the past. I have also been there. When it was too much for me, I quit, and found somewhere more ideal. Now I just keep trying to work with people I trust as they move around in the industry, to avoid all the org B.S. you've rightly brought up.
It's a nightmare that I escaped after three years. I was driven from a happy person to someone who hated his life. It took me a while to realize it was my job. The worst part is my management told us that the on-call pay was already "baked into your salary." I switched jobs internally. No more on-call. Strangely, I was able to keep my pay without having to wake up in the middle of the night any more.
Oh yeah, and you aren't the only one who wakes up. Someone sleeping in your bed may be an insomniac and may have just fallen asleep. Your alert wakes them. Now they don't get to sleep for the rest of the night. Now you and they are both sleep deprived and irritable. It obliterates personal relationships. Maybe your kids overhear you explaining to someone what is going on in the next room. Whenever anyone asks you about your job, PTSD triggers and you spend ten minutes venting.
I am so, so, so glad I'm no longer on call. I don't know what they'd have to pay me to get back on it, but it's at least 2x, and even then, I couldn't last for over a year.
You don't get to go to movies. You don't get to join your friends when they go biking or hiking or camping. You get interrupted in the middle of parent-teacher conference. You have a laptop next to you at the 4th of July party. You get interrupted in the middle of a shower. You don't get to host Thanksgiving or Christmas because you may have to work in the middle of making dinner. You're about the roll the dice to take the first turn of a game you've been promising your kids you'd play with them in the first attempt in a long time to be a decent parent, and your alarm goes off, you work for the next three hours, and they play without you.
There is no 40-hour work week for software engineers. There is no union. You work your normal days then you also get to work normal nights. Then you work a normal day again either fixing the problem that caused the alert or spend 6 hours in post-mortems explaining what happened when instead you should be sleeping.