Breaking Up with On-Call
reflector.dev
reflector.dev
No, for me, the real pain with oncall is that there are a lot of systems in my team. I understand well maybe 30% of it. I'm clueless about 30%. In between for the rest. I can try to fix issues myself (take long time, issues can add up), triage (but means bothering someone else). There are also things than nobody understand, and if they break, it can mean extra days of work for you with extra-stress because you don't know how hard it can be to fix.
Then, some people in the team ship code without adequate testing, because of pressure to ship. Which often adds work to the oncall. So there's all this extra tensions with colleagues which can be hard to deal with for an introvert.
Overall, it "kind of" works for us, but I agree with the conclusion, it sucks. It's really the worst part of my job. I went into software engineering because I like coding. Not because I liked monitor unreliable systems. And I think unreliability is encouraged by management to some extent. That keeps people at work.
Technically it was interesting and challenging, but in terms of stress just not worth it. You could pay me twice my current salary and I still would not go back to it. Now I try to place myself as far away from paying customers as technically possible.
That describes pretty much all of my "full-stack" experience.
What sort of job/background do you have where you are writing low level drivers? I'd love to get into that side of things but I don't know where to start.
I guess hobby robotics, for example, could get you an entry to it if you choose to write the hardware interfacing parts yourself.
In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager).
The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that's what the secondary role is for - to catch you. This means it's perfectly acceptable to go for a run, go to a movie, etc.
You relax the responsibility on the individual and let a sensible amount of redundancy solve the problem instead. Everyone is less stressed, and sure you get the occasional 5min delay in response but I'm willing to bet that the overall MTTR is lower since people are well rested and happier to be on call to begin with.
You also need *ownership*. There is nothing worse than having to support somebody else's work and not being allowed (either via time or other restrictions) to do things "right" so that you're not always paged for fixable problems. Everywhere I worked where the techs had ownership (which varied from OPS people being allowed to override the backlog to fix issues or developers being given enough free reign to fix technical debt) has usually meant that oncall is barely an issue. My current gig I often forget I'm even on call at all and the main issues that do crop up are usually external.
Things like, running in AWS but you have to use a custom K8S install so they aren't dependent on AWS.
Using self managed Kafka so that you aren't dependent on proprietary tech.
It all sucks because they are always less reliable and generate their own errors and noise for on-calls.
If they had to deal with phone calls every time there's a firewall issue that had absolutely nothing to do with the application, they would soon change their tune.
If the primary (paid) on-call doesn't catch the notification, the secondary (unpaid) will be paged. And so on, down a couple more steps, to a senior manager. There's no expectation that anyone other than the primary would actually be available to ack the alert.
That's an unreasonable expectation unless it's clearly said in writing and is billable hours.
Software engineers are an interesting case; some have a great deal of domain-specific knowledge, giving them leverage over their employers. Many less so, and so a union could help. AI might change this equation too.
Replaceable with what, exactly? The local ER is now having to close in the evening because they can't find sufficient nursing staff to keep it operating.
It makes everything cheaper.
What this means is that on call is often "included in your salary" and good luck.
Nobody's going to pay you 1.5 your hourly rate to just sit and wait until something happens. Is that really a thing?
Now if you are called and spring into action, there may be time-and-a-half, if it's outside of your regular hours.
Also most tech companies don't have unions...
> Also most tech companies don't have unions...
In Europe, there are plenty of unions that cover tech people. I'm a member of Prospect (https://prospect.org.uk/).
If there isn’t a union recognised in your workplace, you can build one yourself.
That lasted about 3 months, just not acceptable.
I don't know how much more clear I could make it to you.
Edit: or my UPS friend who told me how the union box loaders would falsely claim alcoholism or drug addiction before being fired so they could abuse the union "protection" that was given to them? Is he trying to dupe me too?
As far as I can tell, unions only show up after decades of management malfeasance. They're kind of a natural reaction. The line "the only thing worse than a union is no union" is probably a hundred years old.
things like more pay/better hours/safer working conditions are appealing to people working low-paid, dangerous jobs but don't really click with most tech employees because those aren't the things they hate about their work.
to win over tech employees unions should talk about more ambitious things like codetermination (i.e., getting workers on the board), 4-day work weeks, remote work policies, employee sabbaticals, etc
If that’s your take away from it I don’t know what to tell you.
I know it varies by situation. When I've been on call I've been able to mostly go about my life. I just had to keep my laptop close, stay in cell signal, and accept I would sometimes have interruptions (typically brief). We fought to keep them infrequent enough that they didn't ruin our lives.
If I want to go meet a friend for a drink or food, I have to lug around a backpack, keep an eye on it to make sure it's not stolen. If I wanted to have a beer or wine, I can't because I may need to work at any point.
Favourite band is performing? I suppose you could take a backpack and the laptop to the venue, but again there's a chance it's pinched, and they'll make you check it at the cloakroom for the performance.
If this is a stated requirement from your employer, talk to a lawyer. This is a common litmus test for whether you need to be paid while on call, even if you aren't actively working. Depending on the jurisdiction you may be entitled to pay (or trigger a relaxation of your company's policies).
One beer or glass of wine renders you incapable? I'd be totally comfortable having 1-2 drinks on call.
https://www.dir.ca.gov/dlse/callbackandstandbytime.pdf https://www.shrm.org/topics-tools/tools/policies/california-...
This doesn't apply to a lot of people reading, but just a PSA for those in CA where it might.
If home is fine then usually all you need is Internet and laptop.
> I cannot go for a run, I cannot go to the movies, I cannot go for a dinner with family, I cannot even go shopping (shopping mall is further than a 10 min. trip)
Sounds more like setting expectations and explaining the situation than a "cannot" (maybe except movies).
You can explain to your family for example that you're on-call and may need to leave urgently. I mean e.g. police do that. It's not that uncommon.
You can take runs within a 10 minute distance back home. The route is up to you. You can start by acknowledging on the phone as someone else commented, which would grant you maybe another few minutes.
There are lots of options. It's on you to workaround it. On-call isn't perfect, sure.
An aggravatingly large percent of them could be resolved by voice over the phone by walking the offshore support team through the same 2-3 runbook items.
"Did you look at the log... I see, OK are you looking at it now? Does it say X? Did you do Y? Good now? Great, goodnight."
"Did you try restarting it.. ok then try that now. Is it good now that you restarted it? Great, I'm going back to sleep"
Ironically we'd have less of these outage calls when the offshore person went on holiday because they'd send one of the competent NY support staff over for 2 weeks. Slept like a log every time.
I wrote about my experience here: https://bobbiechen.com/blog/2022/7/20/being-on-call-sucks
Bug-free software is great, but changing requirements are precisely the reason on-call needs to exist.
Even if you have a whole team of engineers who write bug-free software like this guy, you'll still have failures. Because the world is constantly slipping out from under your assumptions.
Customers never stop changing their usage patterns. They add load at different rates, come up with unexpected requests of all shapes and sizes, and invent new use cases that fly in the face of the original project requirements.
Even if you have created a software system with no bugs that perfectly meets both the functional and non-functional requirements of the project, changes in the state of the world vis a vis customer behavior will come along and change what counts as a bug. If your system has a blanket 60-second database query timeout, and everything's working fine, then there's no bug. But as soon as a new API usage pattern causes certain queries to run on average 10 times longer than before, now you have connection starvation and an urgent bug to fix.
I'm not saying that "timely maintenance and improvement" and "a culture of perpetual ownership" won't have positive effects on reliability. But it's unrealistic that any amount of responsible, careful software development will fully eliminate the occurrence of sudden and unexpected failures. Human on-call, as uncomfortable as it is, will remain a requirement as long as reliability is taken seriously.
FWIW my perspective is that of someone that runs an on-call/incident management platform (Rootly).
The problem is on-call is an essential and critical part of a managerial role, but toxic to those in a developer role.
Managers must be on-call to ensure the appropriate people and resources are brought to bear on unexpected problems that threaten the business.
Developers must NOT be on-call to ensure appropriate attention is spent designing, developing and maintaining the code that makes the business possible.
The rise of software-as-a-service led to companies promoting "devops" engineering which conflates these roles and unfortunately helps unscrupulous executives unfairly squeeze more work from employees.
The core idea of devops, that managers/operators and developers should understand and be capable of performing each other's role, isn't a bad one. Those who understand how the business works at all levels can do more to make it successful. It goes hand-in-hand with continuous delivery.
The best engineers alternate between these roles in a predictable schedule. When in the managerial role they need to observe, react, delegate and escalate problems as appropriate. When in the development role they need to deliver features that create recurring value for the business.
But businesses should not expect engineers to play both roles at the same time!
This form of "on-call" is a toxic moral hazard. It's a sign of instabilty. It's a signal of executive grift looking for a quick pop. "on-call" robs developers of attention they need to develop the features and increases risks that schedules will slip.
It doesn't need to be this way. If a business needs software development it should hire or train engineers with that experience. Likewise if it needs managers or operators to deliver software as a service.
As an operator or manager I look forward to working a shift, but as a developer I will never again accept on-call rotation.
I used to do 3hours rides with my bicycle and go to dinner or social events with a gpd pocket 2 in a small bag.
One place I worked had a 1 in 2 rotation. Every other week on call or weeks back to back if your colleague was on holiday. There was no front-line service screening calls which meant you could be woken several times in one night. All for £30 pcm towards broadband costs.
Most places are more sane than that example but suffer from the same core problem. Follow the sun support is incredibly expensive when compared to putting your existing staff to be on call. Here in the UK, so long as your equivalent hourly rate doesn't drop below national minimum wage and you're opted out of the working time directive (a lot of employers slip an opt-out form into your paperwork implying it's normal to sign it), then it's legal.
Unfortunately I'm yet to find anywhere that on-call operational teams have the clout to get code induced issues high up the priority list outside of cases where they've had to drag developers out of bed at 2am. In my experience that also plays out with getting anything infrastructure based into tech debt budgets. Why focus on fixing problems you don't directly suffer from when you can spend the time on a refactor, integrating a cool new library or spaffing out one more feature in the sprint?
I live in rural Texas. The same things apply here, and more: I'm lucky to have good internet (which enables working remotely) but half my home doesn't get cell coverage so being responsive to a text message or phone call means not even going around my own home (for example, no cell signal in the kitchen means I can't cook while on-call); and with large tracts of land, I can't go out to do land maintenance (good luck hearing a phone ring or feeling it vibrate from a call when you're operating heavy machinery, assuming you even have cell signal there); all services are 15 minutes or more away: groceries, doctor, contractors, government, etc etc.
It's important to stress how much being on-call ruins my capability to use the time effectively for my own purposes (Texas Guidebook for Employers [0]; 29 CFR 785.16 [2] and 785.17 [3]). I tried telling this to a previous employer when they started wanting me to be on-call (3+ years after start of employment), and they indicated that those laws are only used for hourly employees but being salary + exempt means I do not qualify for additional pay and falls under "and other duties as assigned" in the employment contract. So the employer effectively started getting 60 hours of work for 40 hours of pay. Oof.
I also absolutely refuse to mix my personal devices with work; just at a minimum, I refuse to make my personal device available to legal discovery related to any legal issues with the employer. So if the employer wanted me to have cell phone availability, then I demanded that the employer provide that cell phone. That was a fun conversation that ended with some relaxed requirements (eg, I don't have to have cell phone availability if I'm responsive at my work desk already) which further reinforced the fact that I couldn't use the time for my own purposes.
Thankfully multiple years in this industry at (what was) fair compensation allows me to be picky for new employment contracts. And lesson learned: I'll be a lot more careful about contract language from now on, and specifically look for (or negotiate) carve-outs around being on-call and work/personal device separation. I recognize that having 10+ years of experience makes me able to handle that, but newcomers to the industry won't yet have that buffer and it sucks for them to not have that safety net for negotiation leverage.
A lot of this disagreement comes from businesses demanding rapid response while insisting on not taking on new hardware/payment obligations. To contrast: take the fireman who's waiting for an alarm (29 CFR 785.15 [1]): they are often often idle and can often go out for groceries but they're easily reachable. Ever seen a firetruck in front of a grocery store and the firemen are just inside shopping for groceries? Then see them come running out and turn on the lights & siren and drive off? I have. It's an interesting event, and it sucks for the grocery store that has to put those groceries (for ~15 people) back on the shelves and refrigerators. Nonetheless, those firemen are paid to do so and have special equipment (eg radios or cell phones) to be able to receive those messages, and the firement generally don't pay for that equipment themselves (the community does either through taxes or donations). I see analogies about on-call software engineers being called to put out (virtual) fires as very apt in this case.
[0]: https://efte.twc.texas.gov/c_waiting_or_on_call_time.html
[1]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...
[2]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...
[3]: https://www.ecfr.gov/current/title-29/subtitle-B/chapter-V/s...
When engineers bear the full financial consequences of 3AM pages, they're more likely to make systems more resilient by adding graceful failure modes. When incident response becomes an organizational checkbox divorced from financial outcomes and planning, you get perpetual firefighting.
The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated.
Big companies aren't missing the resources to fix this; they just don't have the aligned incentive structures that make fixing it rational for individuals involved.
The most rational thing to do as an individual on a bad rotation: quit or transfer.
If it's important enough to deserve a page, it's top priority work. The reverse is also true (if a page isn't top priority, disable the paging alert and stick it on a dashboard or periodic checklist)
if it's just the engineer while product and management see no real cost then people burn out and leave.
> The most successful teams I've seen treat on-call like a leading indicator - every incident represents unpriced technical debt that should be systematically eliminated. Each alert becomes an investment opportunity rather than a burden to be rotated.
100%
Making engineers handle 3 AM issues caused by their code is one thing, but making them bear the financial consequences is another. That’s how you create a blame-game culture where everyone is afraid to deploy at the end of the day or touch anything they don’t fully understand.
A manager no longer needs to choose between system reliability and churning out new features with on-call:
The manager can get all the credit for pushing out new features during the day, and sleep well at night knowing that the engineers aren't.
Besides, if you set the on-call system up so people get free time the following day to compensate for waking up at night, the manager can't pretend there's no cost.
Bad management will fail on both of these of course, but there's no saving that beyond finding a better company.
Support work is a necessary nuisance, but it's also not what on-call is meant to be.
It's normal in any kind of system to also cover issues.
Yes, at some point you might want to have a customer support / customer success later to at least triage them, but that makes more sense as you get bigger not when you are small.
I actually like having discussions on support days with customer. Yes, sometimes they're more annoying but it's direct feedback of people trying to use your software.
That was the better part of a year in my life, continuous. Of constantly considering that every decision, every meal, every movement, every action at all times, and weighing it against the risk of impacting my ability to respond to a customer call. Maintaining an extended period of alertness for a threat that very rarely materializes is frustrating in many ways that I'd like very much to forget.
I didn't burn out from it, but it was a major factor in my decision to resign from that company. Obviously people out there that can handle this lifestyle, but I couldn't. And frankly I'm quite content to never try again.
No I don't think anyone could. What you're describing is an insane policy of abusing employees. Most on-call is done on a rotation, not permanently.
All of my previous roles involved on-call and it was 1 week per quarter.
It was fine, but as an SRE/DevOps person I had the authority to override the developer backlogs to get issues fixed and enforce monitoring, architectural, and quality standards. Technical debt rarely built up and we could take one step back before taking two forwards. The product people hate me at first, but the result was usually a platform where problems only ever occurred during deployments (which I've been mostly lucky that I was at places we could do it during business hours) or external factors happened (cloud resource issues, etc).
At my current company, I can count on one hand how many times I've been paged out of business hours in the last 4 years (we recently re-did oncall because it was recognized that it was a risk that I'd become irreplaceable, which is the one point I tried to hammer on them for awhile).
The worst on-call experience I ever had was working at a managed services company. That was the hell of supporting customers that refused to properly allocate resources or invest in their tech stacks. Never again.
When you have on-call like this, you offload your expenses to your employee's life and mental health, and it's cheaper on the paper, for the short term.
Then your company sinks as people starts leave, and the managers ask "Why?".
Easy to say when you envision it as someone's office hours and not your office hours. If my employer gave me a choice between a normal shift plus oncall or a night shift with no oncall, I'd pick the on-call in a heartbeat.
It also meant having to wake up on command in case the phone beeped, being unable to drink in my free time, being limited in which activities I could go to.
It was hell.
I know people are gonna hit back with "you're doing it wrong", but in this case it's the company doing it wrong, but nobody on HN will go there and tell them.
What kind of slave driver expects literal 24/7/365 availability? Does that not breach labour laws where you work?
Unfortunately this is a huge hole in German labour law.
I quit there pretty fast, after a couple months. It was a tourism company, so Corona treated them very well.
The other company where this happened was a Content Marketing company that was wiped out by ChatGPT. I didn't do on-call there but the other team did.
So were you the only one ever on call? That's rough. I've been in two man (every other week) rotations and that was annoying, but went to being solo on call for about a month. That was a shitty month, despite not having many calls. It's just the constant thought and consideration like you mention.
> The same friction applied to fixing a bug or a flawed process. I would reproduce the bug: spin up the entire environment, the appropriate binary artifact and the reproduced state of application, create the test cases and pin-point the exact problem for the stakeholders as well as present the possible solutions. I would get sent to a dozen of meetings, bouncing my ideas back and forth until receiving the dire verdict - rejection to fix this bug altogether.
This has been my experience for the entire duration of my tenure at the current place I am employed at. I've stopped doing this, because my backlog is filled with "best effort" features and when I attempt to slot some of these into a quiet sprint, management says no.
Many features I request is not from assumption, but data I gathered with analytics for the customer-facing website. One example is to change a page layout and add better search functionality for this page and its data, because I noticed a +70% no-results rate in the search analytics. I suggested a change but marketing shot it down saying it's low priority for them, while they frequently say that the website can perform better at generating leads. I might be wrong, but to me it feels like utter short-sightedness and goes against the strategic goal of the company.
I'm just an engineer, what do I know?
It’s a low stakes change.
Nobody assumes harmful intentions from the author - I would not have recognized the picture either. But now that it’s been pointed out that it’s from a site where people were held illegally against their will, the reaction is a tell-tale. Knowing this, and insisting on keeping the images is now willfully associating with harmful behavior.
Apart from that, I cannot associate with the picture either - as an on-call engineer for some widely used infrastructure, I am not a guard on duty keeping people in a camp. I am an emergency responder. I fix things when they go haywire or an accident happens. A firefighter, paramedics, civil emergency responder would IMO be a much better metaphor for what I do.
Then what's the issue or purpose? Other than satisfying or inflating some own moral selfimage? That kind of social dynamic appears to be what this kind of things is about most of the time as opposed to preventing mental harm or stopping such issues from (re)occurring.
>Knowing this, and insisting on keeping the images is now willfully associating with harmful behavior.
Do you not associate with harmfull behavior in far far more direct ways? Perhaps something that's considered ridiculous for most people to avoid like paying taxes that get spent on bombs or the like?
If i want to make an article showing a prison and i pick one from google does it matter if that prison happened to be used in the past to house civil rights activists? Is the implication that that somehow normalises, advocates for or otherwise inches towards oppressing civil rights movements in any meaningfull way?
To not use a thing that symbolizes the indiscriminate incarceration of innocent people as some clipart for an article about oncall. It's in bad taste, and also the wrong metaphor, and it sends the wrong message, even if not done on purpose.
> Other than satisfying or inflating some own moral selfimage?
Morality has a subjective component and often works by seeing through other people's eyes: How would I feel if this happened to me? How will other people see me if I did this? There's no universal consensus for this, of course, thankfully.
So when people disagree about morality, I've found that it can be hard for the side that doesn't see a problem to understand what the other side is making such a fuss about. I also think there can be a performative aspect to moralizing. But I don't think it's fair or warranted to jump to the conclusion that just because one doesn't see the issue, that the other person must be posturing - because that's how you can make sense of their behavior from within your own point of view.
People see things that rub them wrong, they speak up, as did you. How would you know you're not just inflating your self-image by putting down their comment like this? If you aren't, maybe they aren't, either.
If you truly wanted to do some microscopic good you could've just used this picture to quickly bring awareness to what happened and could happen again rather than try to relegate the story to the view of those interested in history for the benefit of....
1. That it's harmful
2. That it's easy to do something else
(1) is the important point, (2) is irrelevant. On the important point, you seem to have assumed several things:
a) that it is harmful
b) that anyone to whom the assumed harm would occur would see it
c) that they would know what the camp looked like
(a) I see no evidence for this, in fact, research suggests the opposite[1]
> The strange paradox about triggers and PTSD – and this is true for all anxiety-related disorders – is that avoiding triggers makes the disorder worse, not better (Jones et al., 2020). Being exposed to small instances of one’s triggers, in a safe environment such as therapy where one can be helped to process the situation, is a way to gradually become less reactive to those triggers (APA, 2013). Many people have learned to reduce their reactivity to psychological triggers through this process, called exposure therapy.
and (b) really is a stretch, are we to believe there are people in the tech industry who are on-call and were interned in WW2 and will read this article and will have some kind of meltdown because of it? Why would their descendants react badly to it? Unless they have a pre-existing condition, that's untenable, and if they have a pre-existing condition then they should seek help for it.
And to (c), how would they even know what the camp looked like if they're so triggered by the thought of it? They have an aversion to it.
No, none of that makes sense, and cloaking it in "compassion" or trying to handwave scrutiny away because it would be a "low stakes" behavioural change won't hide it.
> Apart from that, I cannot associate with the picture either
We are intelligent people, the link is really simple to make. There being "better" choices does not make the picture a non-sequitur.
I'm sure I'm not alone in wishing this West coast American style of pop psychology being misapplied to real life would die a death.
By the way, George Takei (I believe that's who you meant) is 87, and he's not really in tech.
But okay, I really want there to be a picture of a concentration camp guard tower, because the best way to deal with childishness and unreasonableness is, unsurprisingly, childishness and unreasonableness.
I also couldn’t take it seriously when article opened with this.
> Startups cannot afford engineers to baby sit software, big tech does.
Say what? As ops person, I’ve seen multiple startups where devs are drowning in operational issues because software was written enough for feature MVP, ship with poor testing then constantly poke at with sticks to keep it working in hopes they could get enough revenue to not flame out.
Engineering rarely matters unfortunately but ability to weave a story and sell that story.
An ad for their consultancy. It's a call to action, not a balanced analysis.
It seems like lack of automated remediation would be a bug unless it's an "accepted business risk" i.e. cheaper to throw people at to manually fix than build a software solution.
First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case.
Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system. This is obviously not the case.
Third, and finally, a system failure addressed by someone "at 3am" assumes said person can rectify the problem in real time and without any coworker collaboration nor stakeholder input.
The last is most important:
If a resolution requires a system change, what organization
would allow one person at 3am to make it and push to prod?It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years.
While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure).
So this may end up being a cultural difference between on-call situations.
That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe.
> While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure).
My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support.
If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same.
Just my $0.02
If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.
I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.
Pain from a problem needs to live where the problem can be fixed.
This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot.
> Pain from a problem needs to live where the problem can be fixed.
Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].
If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience.
> Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].
It's not punitive management. Someone will have to work when the thing breaks. It can be the people who can fix the problem, or it can be someone else.
A couple points come to mind here.
First, there is no reason for any employee to have to "wake up at night and fix it" if the organization staffs SRE's with a "follow the sun" model. Three complementary time zones are ideal, but two can suffice if one is first-shift and the other second shift.
Second and more importantly, splitting responsibility between a "dev team" and an "ops team" is problematic when the two teams do not communicate, coordinate, and assist each other in improving the system. Hence this situation qualifying as a leadership problem.
> If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience.
A and B are different jobs, with different responsibilities, with different skills, and different success objectives. Combining the two is a recipe for burnout and/or failure.
>> Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].
> It's not punitive management. Someone will have to work when the thing breaks.
The statement I quoted was:
Pain from a problem needs to live where the problem can be fixed.
If A and B are the same people, very little leadership is
needed as they are innately motivated ...
If that's the case, then the person responding at 03:00 needs to have a serious conversation with the dev, or with whoever's assigning/prioritizing tickets to the dev.
That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org
I completely agree.
The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly.
Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.
Well yeah, if you have on call, I guess you have to trust the on-call person to make that decision. If they get called at 3am, the problem is obviously so bad that almost anything they do can only make it better, right? If it's some overlooked edge case that crashes the whole application and can be fixed with a one line change, it's probably Ok to do it a 3am. If it's more complicated, maybe wait until 7am and discuss it with others too. And, in any case, a thorough post mortem (during office hours) and checking if the 3am fix is really a proper fix for the issue is in order...
The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations.
When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.
As I mentioned in a peer response:
Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].
> If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?
The only advantage of having a micromanaging manager: if they want to review all PRs and system changes, they also must be on call every single night of the year.
Because Amazon oncall is ... well it's something. But I'm not sure it's really indicative of the rest of big tech oncall.
One big problem is that they tend not to make any attempt to ensure that the on-call rotations are equitable across teams. You might need to be on call one week out of 4 or one week out of 20. At review time, this matters. Since you're typically not getting paid for the on call work or having it factor into your performance review, you're actually incentivized to snooze alerts and kick the can down the road until after your rotation or to join another team with lower rates of on call incidents.
Ironically, if you're spending 25% or your time on call and actually being diligent about the role, you're more likely to be adept with the typically hodgepodge collection of tools that you need to use to diagnose and address problems and will be better able to handle incidents than if you only ever use those tools once every 5 months.
I had just been promoted to a systems admin role, after demonstrating prowess with coding and system configuration, and also the actual qualified admin had quit. One of my projects involved a rewrite of server code for efficiency as our /etc/passwd balooned to thousands of users, and too many logging in simultaneously.
So on Thanksgiving Day, we just got in the door with Mom after shopping, and my supervisor rings the home phone; his first words are "There's a hacker on [production shell server]". And I told him a few commands to try, and we did not seem to suffer downtime or need our backups.
That may have been the most consequential job I had for a decade. There were thousands of customers depending on us giving "four nines" of available service, including the corporate customers, but we had no formal on-call agreement, not even a way for me to access systems from outside the office, and I was still a novice after a few college courses and a lot of unstructured hacking.
My [salaried] father, on the other hand, often brought a pager [which never seemed to beep] home, and one day we were all sitting around the living room when an explosion rattled our windows. We were still a bit rattled ourselves when the phone rang, and it was his cue to go down to the site and respond as a health & safety officer.
So my mother said that our household policy would henceforth be: "if something explodes out there, nobody answers the phone!"
So if you had family that needed your immediate help, they would not receive it?
That's indeed a perverse incentive.
The central controversy of on-call duty is that it leaves the household and family short-staffed, contributing to burnout and "workaholism". We were accustomed to Dad being at work M-F, 9-5, and being available and supporting us at all other times.
While a boiler explosion is certainly exceptional circumstances, it can be the "camel's nose in the tent" and eventually, Dad is potentially driving up there every Saturday night that a lab worker spilled 1 gram of mercury. Then he's not eating dinner with us, resting, mowing the lawn, helping Mom to shop or do errands.
I'm trying to recall any times Dad responded to a family emergency by leaving work, but it was nearly never necessary, since preparedness and response plans defined his career.
I don't work for employers who can't manage their staff properly. If you're on-call at a company with any sort of scale, you aren't being managed properly. One of the things I tell hiring managers is that I don't mind working later if we're reaching the end of a project and a late afternoon or two might get the product pushed out of the door sooner, or if I've gotten into the zone and am making reat progress. But these are my choices to make. There will never ever be a time when I will subject myself to being tied to a phone for the purposes of waking up in the middle of the night and messing with servers or software. My time is MY time. I only ever have reciprocal relationships with employers and refuse to upset that balance.
https://www.eurofound.europa.eu/en/european-industrial-relat...
I have seen it vary across companies:
- Some companies don’t pay any differential
- Some companies only pay out _per incident_
- One company I worked with paid out per shift of on-call rotation
- One company treated it as overtime pay
This is quite scummy unless your role's primary duties are support and nothing else. I would count Site Reliability Engineer as support being a primary duty, while I would not count Software Engineer as support being a primary duty.
> Some companies only pay out _per incident_
That's a bad incentive. It incentivices more incidents to occur.
> One company I worked with paid out per shift of on-call rotation
Sounds decent on paper but everyone has their breaking point!
> One company treated it as overtime pay
Sounds decent on paper but everyone has their breaking point!
His experience has nothing in common with my on-call experience working at a different "big tech" company.
Fair deal, I'd say. Only problem is that most of the applications are from vendors, so getting problems fixed is an ordeal.
In this day and age if you're a properly capitalized startup you should have the budget for one decent SRE hire. They often times double as a technical writer and are great at preventing knowledge silos in teams / members of a small team since they have to sort of know everything to be an effective SRE.
Best process improvement I ever made for my teams.
> Creating an on-call process to manually inspect errors in test suites is more valuable than improving the project to be more reliable, as you can directly measure the amount of tests that failed on a weekly basis. It is measurable and presentable to the upper management.
You can also measure requests that failed on a weekly basis, and I do. In fact, I added a dashboard panel to do exactly that today for a service (10 years old!!) on a new team I just reorg'd into. I did this because I was annoyed to discover the first (internal) customer outage report of the day could have been repaired by the east coast half of the team team hours before the west coast QA team logged in for the day, but they were unaware anything was wrong. This is a trivial promQL query to implement, and yet it wasn't until today.
The problem isn't visibility but risk -- what if you make reliability fixes but the data gets worse? This is not hypothetical, a Youtube engineer documented a similar tale[1]. You can also imagine all kinds of fixes that sound good on paper but can produce paradoxical outcomes (i.e. adding retries causes a metastable failure state[2]). And heck, what if you make no changes, and the numbers decline all on their own? Are you going to scuttle this quarters project work (and promotion fodder!) just to bring this KPI back to normal? Of course, all numbers, even the test suite pass rate, come with risks of missing targets, so the incentives are to commit to reporting as few of them as possible.
> tools automate the mundane tasks of an on-call engineer: searching for issues related to a customer report, tracking related software (or hardware) crashes, verifying if the current issue that arose during an on-call is a regression or a known bug and so on.
I have a coworker trying to use LLMs for ticket triage, but there's a huge GIGO risk here. Very few people correctly fill in ticket metadata, and even among the more diligent set there will be disagreement. Try an experiment: pick 10 random tickets, and route copies to two of your most diligent ticket workers. Then see how closely their metadata agrees. Is it P1 or P3? Is the bug reported against the puppet repo or the LB repo? Is a config change feature work, bug fix, or testing? Do they dupe known issues, and if to, to the same ticket, or do they just close it as a NTBF known issue? If these two can't agree on basics, then your fine tuning is essentially just additional entropy. Worse, you can't even really measure quality without this messy dataset, and the correct answers should change over time as the software and network architecture evolves.
[1]: https://blog.chriszacharias.com/page-weight-matters [2]: https://www.usenix.org/publications/loginonline/metastable-f...
I will admit, though, the SaaS tech industry as a whole has a problem. The cargo cult of tech has convinced itself that SaaS is special. That it's never done. That it must be never finished, never stable, always changing, always breaking. Like it's some immutable law of nature.
This of course is a convenient lie. Before every business was an online service, software worked like every other product in the world. You built it to do a thing, you tested it thoroughly to ensure it didn't have defects, and then you coordinated one large release. No constant changes or late Friday deploys. No schema changes destroying columns. There was no on-call, because there was no 24/7 service. There were floppy disks and CDROMs, and people running software on their own computers, and a very small number of 24/7 enterprise systems shared by lots of users (mostly run by ISPs, and one or two large tech companies and industry bodies).
But you can deliver online software like it's desktop software. You can ensure your software is bug-free before you burn your master disk. But then you don't get to skip all those time-consuming tests. Then you don't get to ship your features faster than your competitor. Then you don't get to do A/B testing and micro-tweaks and experiments and feature flags and blue/green deploys. We want to do all the things, at any time, with no consequences. So on-call is still a thing, because people want to have their cake and eat it too.
The actual infrastructure on the backend shared by millions of users? Not actually hard to maintain. It's the same shit as 20 years ago, but much, much easier. Make sure the disks don't fill up. Auto-scale the VMs and containers. Design your apps to not exceed network bandwidth. There's really not much else to break. The only thing that breaks a system is changing it, or bugs from not enough testing. So do the testing, and the bare minimum of autoscaling in the cloud, and nothing should ever break in production.
But constantly mutating SaaS, without due diligence, is too addictive. Nobody's going to abandon the freedom of doing whatever the fuck they want, just to keep from waking people up in the middle of the night. The business people sure as fuck aren't going to abandon their "competitive advantage". And devs who don't care if their software works or not don't want to abandon untested deploys from their laptops. So on-call is here to stay.
That said, my last workplace completely burned me out with a terrible on-call policy and an absurdly short recovery period. Not to mention, upper management tried to gaslight everyone into thinking on-call was just part of normal work and didn’t warrant additional compensation.
I get why it’s necessary, but I’m lucky enough to be in a position where I can flat-out decline any job that mandates unpaid on-call.
You can have a system with enough redundancy, ability to rollback and deployment scheduling where any sort of on-call incident is highly rare and low impact. But that requires spending time and money on solving those problems which is money that could be spent on developing more features faster. A buggy and broken product doesn't matter as long as you can shovel out More faster.
I've been on both types of teams and I'm at the point in my career that if I'm going to be on-call I'm expecting to be compensated appropriately with actual on-call hours worked. And part of the problem is that even on a team where you're informed of on-call duties and rotation, if the team gets cut or people leave then you're on the hook for working longer hours for less pay. It's inherently exploitative.
Fail-safes are one thing, but they don't always kick in when we need them. My team is only paged one or two times a month, and when we are it's probably for something we've not seen before because whatever it was that broke last time has been fixed.
Some companies might have exceptional on call - the C suite will get called back from holidays or whatever should things really hit the fan. That’s once every few years.
It is part of normal work for roles where it's a thing, and I'd think it could be either baked in or on top as long as which one it was was known at salary negotiation time.
Every position I've held (save my current one), that is most definitely the norm. If you have a busy night managers are cool with you coming in late the next day (or potentially not at all), but it's very unusual to be paid for on call in my experience.
If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations. If you knew about that going in then you weighed it against the offered salary and decided it was worth it. It's not "unpaid on-call" in that case, it's paid on-call.
Obviously it can also be done poorly in ways that are harmful and dishonest. Not communicating it clearly during the job interview and salary negotiation, applying it inconsistently, or changing the frequency or the difficulty of the rotation after you have started are all real problems.
You'd have to pay me a biblically large multiplier on my base salary to get me to pick work over my kids or wife. My kids aren't going to remember that I made a few thousand bucks by ducking out halfway through an important-to-them event. My wife is sure as shit ain't going to forgive me _or you_ if you call me into work in the middle of some adult time. And the cost for you to make sure that I'm not participating in such activities and am available to work on very short notice is...well, it's the exact same number as if you had interrupted such activities.
If you really need the coverage in the middle of the night, hire more engineers whose normal 8-ish hour shifts are during the night? Can't afford to? Perhaps you're not so good at business then. Better yet, stop building a fucking house of cards that can topple your company if it wobbles a little bit. If your systems going down for a few hours can take down your company, outside of some exceptional circumstances, that's a failure on your part.
> If it's made explicitly clear in the job interview process that on call is expected in a role and if it's enforced evenly across the organization, this can absolutely be normal and ethical—you're on a specific salary and your salary includes on call expectations.
Where this logic breaks down is when on-call expectations are in addition to what a salaried position compensates - a 40-ish hour work week.
Expecting an employee to be paid industry rate for services rendered and then expecting periodic 24-168 hour near real-time availability without compensation is, by definition, "unpaid on-call".
So if on-call isn’t explicitly compensated, an employee quitting essentially gives the rest of the team more hours at a lower hourly rate.
We all know it. You trade your peace of mind, sleep, and sometimes personal freedom (hello, “don’t go beyond cell signal” life). And I agree with many of you—it shouldn’t be this way.
We’ve seen teams turn their on-call rotations into something way more humane. Not perfect (nothing in ops is, let’s be real), but sustainable.
At Zenduty, we work with a lot of SREs, DevOps folks, and incident commanders who’ve said, “We can’t keep doing this.” So they started rethinking on-call. One of our partners cut their alert volume by 50%. Half! That’s not a magic trick. Just better processes, better tooling, and some good old-fashioned empathy for the humans behind the screens.
And if your team’s not giving you breathing room after an all-nighter? You deserve better. I’ll just say it.
I saw some of you mention how this stuff becomes political or gets deprioritized in favor of shiny new features. Ugh. I feel that in my bones. The best teams we’ve worked with treat incidents as opportunities to make things better. Every alert is basically your system saying “Help me, I’m tired.” And when you listen and fix it? Less 3 AM wake-ups. Win-win.
Anyway, I didn’t mean to ramble, but this topic is close to home. If you’re still stuck in on-call hell, or just want to vent about it. Always happy to swap war stories or chat about how folks are making on-call suck less.
Stay sane out there, Rohan DevRel @ Zenduty