I Don't Want to Be on Call Anymore
honeycomb.io
honeycomb.io
What some people struggle to realize is how easily the responsibility of an individual can scale at certain companies, without their compensation improving in the slightest. If you are working on-call and not making big bucks your company is likely milking you dry.
It seems these days to just mean "multiple millions", which reduces the utility of the term.
Having a few billion gives you some additional fuck you privileges - police officers, world leaders, other billionaires -, but you probably won't get on a level where you can say that to anyone and not fear repercussions. Also, if you can live happily on your own, you can already tell them that if you have a few million to your name.
Most will not accept these terms, and that's the goal.
Forcing them to think in your hours can also have the nice side effect that they get their communications in order.
If you don't get 10 hours continuous break before the next work day, you fall into fatigue leave and either get paid overtime for the next days work or are stood down till you have had a 10 hr break.
Still doesn't stop our operations team being completely snowed with both workday and oncall work, but they can make a fairly good bonus wage out of it.
Minimum call out fee of 2 hours regardless of the length of the call.
Also in Australia. This only started when I joined as an SRE.
Prior to that the Devs just said no it'll stay broken till we come back to work and that was largely acceptable. The on call rates only started as our reliability became more of a business need.
One of my reports pointed out that he made more from on call each year than the difference between an average and outstanding performance bonus. He wasn’t wrong.
You get paid 50% of your normal hourly salary for every hour you're "available".
You get paid 200% of your normal hourly salary for every hour you're working on an issue, incident, outage, or problem.
So being on-call is pretty lucrative, given that you'd usually do it for a seven day period. (i.e. 9am Monday to 9am the following Monday.)
Both of those are very relative. I've seen on-call which means daily events, and on-call which requires something once a year. There's a very different level of compensations I'd be happy with in each case.
Yeah this is the right answer. I know a lot of people who are earning more than I am in exchange for being on-call (or being de-facto on call, ahem) and it just ain't worth it.
If an organization thinks their systems should be available 24x7 they should staff people 24x7.
There is no situation where it’s ok to contact someone to do work outside of their work hours.
Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.
If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.
Salaries in the industry have a lot of expectations baked in, like the expectations of being able to learn and train yourself without the company having to pay for it (aside the occasional paid book). Being on call is another. That some folks can get the same pay regardless of their responsibilities or how hard their job actually is is, indeed, a problem.
Let's take this idea to it's logical extreme then. Surely you would agree that slavery is abusive. If we imagine a society where there are only slaves and slaveowners, your position would lead one to believe that slavery is not abusive, simply because it's the status quo.
This practice is abusive because it subverts the agreement that one will exchange their labor for pay within a defined set of hours (8 hours per day) and replaces it with the expectation (not agreement) that one will be "available" 24/7, but not actually "working" unless a pager goes off. This is plainly abusive, because it destroys your ability to use your free time to do things like drink alcohol, smoke marijuana, go hiking, go sailing, go for a run, etc. because you are required to be online with 15 minutes notice.
But for salaried employees, this isn't the agreement. I sometimes work four hour days. Sometimes I work odd hours. The agreement, for salaried employees, isnt about hours, it's about work getting done.
If someone consents to that, swell! I think many companies are abusive in that they don't compensate oncall work enough, but salaried oncall positions aren't inherently more abusive then having a lawyer on retainer.
My previous job I was on-call 1 week out of every 6. My first on-call week I got paged 100+ times (obv many of these alerts were firing within a minute of each other, but even accounting for that I was still being woken up multiple times per night). That week was particularly bad but I was still woken up an average of >= once per week on my on-call weeks. My team - the sole SRE team - was on-call for everything, including of course the 6 other dev teams' services.
Even worse, when I would fight to get bullshit alerts removed (the alerts that don't actually indicate a problem), I would get incredible pushback from, of all people, my own manager! (who was the manager of the SRE team since like I mentioned we were the only SRE team)
So not only were half the alerts bullshit, but I was counter incentivized to actually fix the problems causing the alerts. One great indicator of how bad things were was that everyone else on the SRE team (except the manager but he somehow always had unusually light on-call weeks) had a 5 minute delay set for pages, because the vast majority of pages would resolve within 1-3 minutes.
Oh, and by the way we weren't given any extra compensation whatsoever. Indeed I was told it was "part of the job description", which effectively meant that SRE skills were less valuable than the devs, because the devs were making identical salaries with no on-call requirement whatsoever! So a more specialized and, at least at this organization, difficult skillset, was worth less.
Anyway, I quit that job, and never looked back. I still have a friend on that exact team who is still putting up with being on-call, despite the incredible impact it has on his ability to go out and do stuff (mid 20's guy). I hear him complain about it all the time. But, like most people, he doesn't have the balls to either (a) push for internal change (which in fairness I tried and failed, although half the reason I failed was because nobody else on my team was willing to stick their neck out and push for the change with me), or (b) quit and find a new job.
So...yeah. Kind of rambling but we have a huge problem in our industry with people who either "don't have lives", or kind of do yet have so little self esteem or whatnot that they can't actually say no and protect their personal lives. In most cases on-call is simply a case of someone getting a raw deal and being too afraid to admit it to themselves.
Honestly, I'm not even sure how it's even remotely possible to draw a parallel between slavery and that. When negotiating a salary you'll have to take these things into account. Not worth it for the pay? Find another place to work. Jobs do grow on trees for people in our industry.
The question on whether it's abusive or not very obviously depends on the details. Yes, lol, I am actually willing to accept $250k per year, unlimited time off that I use upwards of 6 weeks per year of, full benefits, working from home, in the event my boss needs to call me once a year on a Saturday.
Sweeping generalizations are not helpful, and will likely hold you back in your career. You should evaluate things on their own merits.
I have bootcamp students with 6 months of software development experience making more than top developers in France, lol. The few extra Euros you get for being on call hardly seems worth it.
There's a reason French workers are almost always rioting. Low pay and very few opportunities.
No place I have ever worked has this kind of oncall policy.
Most I have seen have a response time of 1hr, note resolution, but response.
Amazon is large enough they should have people staffed 24/7 just by staggering the timezone codes at the different offices. This is how Cisco TAC works, you call them you get what ever timezone code is "day time" at that time.
So I would agree that 15min on call is abusive, luckily this is not my experience as "typical" in the industry, hell even fully staffed 24/7 call centers typically do not have a 15min response time
By working for free you’re subsidizing your organization’s unsustainable business model.
Don’t like it, quit and find a company that is moving slowly and doesn’t value ownership from engineers, someone else will fill in the vacant post. Sounds like a reasonably sustainable business model to me, especially since it has worked like this for over decades at some companies.
This has totally not been this way for decades.
Now if you're a founder, or you're being paid for this extra work or own a significant part of the business and benefit from it's success, that's a little bit different.
The lawyers at my company go home at 5 and make way more. I know a doctor who gets paid somewhere around $1500 for every day he is on call. And when they actually call him he makes even more.
Of course, you can still go to law school if you're really that jealous.
That said, there is a level of seniority at which "their problem" is "your problem." If you'd chuck the business out the door rather than getting paged a few times a year, you're not ready for that level yet. This varies by company and organization size, but fundamentally you can't (and shouldn't try to) anticipate everything up front. Sometimes you need the knowledge, judgement, or simply signing authority (literal or metaphorical) of someone specific.
I’ve long decided that unless I have the significant stock options to show for it it’s delusional. It’s what the company wants you to think: to have all the emotional investment and none of the actual ownership benefits.
If you want a cushy 9-5 job, go work for a code factory like HCL, TCS and whatnot. That way you can build crap and wash your hands after handling it to the customer.
In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
I've been in oncall roosters several times in the past. And it made freaking sure we wrote the best software we could.
I also have been on the management side of it, and the way I set it up is that whoever was the oncall engineer, if he had to work for more than 2 hours after normal working hours, he would get an additional PTO day. It actually worked quite well.
You got it backwards: if you want a crappy quality of life (or have no life and you only identify with the company that sees you as a replaceable cog), and crappy products and customer service from a company that doesn't care (to plan right, to hire accordingly, to treat its staff right) go on call and overwork, producing sleep-derived crap.
Craftsmen and artisans take their time and have boundaries.
"Feature factories", startups, and mass market crap companies forego 9-to-5 and indoctrinate naive employees that they do something important by doing so.
>In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
Yes. It's for starry-eyed naive youngsters right off the bus. The kind of people to believe they're "changing the world" by building a Facebook or Groupon.
I've been an engineer on-call for 7 years since graduating. In the meantime I handled a handful of business critical situations, developed my ability to keep cool during crisis and put my skills to the test. When I started doing it, I wasn't even paid for on-call. Now I am. The money has no bearing on me having been exploited or not. I've done it because it's cool, and when I think it's too tiring I won't do it anymore.
Am I less of an engineer, not a true craftsman (wtv that means), because of this? There's other opinions in the world, no need to be so close minded.
It's also the kind of theft that insidiously convinces you that it's really cool and that you're actually OK with it because you're learning how to 'keep cool during crisis'. You know how else your employers could have taught you that? By providing actual training during working hours, while you were being paid.
It's fine to drink of the chalice of the corporate kool aid with moderation. Still, one should never get black out drunk out of it.
We can't just chunk these complex systems over the wall and say "have fun with that" and demand more money when the solution breaks. This means we failed to properly estimate total cost of ownership when we sold them the system and now they're held hostage. This is borderline legal breach of contract.
If we told them the true sustainable cost of ownership for a follow the sun support model they likely would have never purchased the solution in the first place and we'd be out of a job.
We could have lots of annoying jobs or no jobs at all. I'll happily accept the former.
I'm not begging anyone to let me prove my technology, I'm working with management to come up with a solution that will work for the customer. Oftentimes management will forbid or otherwise prevent me from using a better, more robust technology that would have prevented more production issues because of short-sightedness. If I pointed this out and was still told to go ahead with the short-sighted solution, and it breaks in production, whose fault is that?
I'm not suggesting the 'it's your problem now, have fun' approach, I'm suggesting the 'pay overtime for overtime work' approach. This is nowhere even near to 'breach of contract', please don't throw around lawyer-like terms when you clearly don't understand what's being discussed.
That's great, but you could still have done all those things if you were being compensated appropriately for the extra time you were putting in.
It's like unpaid internships. I'm sure unpaid interns often do learn useful skills and make useful connections. But the practice is still exploitative.
Convincing them of the "coolness" of it is the most common way to exploit the naive/fresh.
The same people who do so to others, wouldn't even piss if they weren't compensated for it...
(And being exploited or not is not a personal decision. If you aren't compensated for overtime, then you are exploited. It just means you're ok with it.).
I had that too, but it was for art projects, films, music, my own programming projects or some other thing that I certainly did not do for money.
The one thing that differs here is that the people I did this for/with would cross the country if I told them I am in need.
they had a laser focus on just shipping and success theater, but little attention on actual utility of the end product. i was berated constantly for slowing things down or wanting to be more careful and do things that weren't necessarily easy but actually had potential to be useful. it's relieving to learn this is a common management deficiency.
Yeah but most of the people complaining only think they are Craftsmen but their day job is to write bad software to push ads
Let it crash. If you can't afford to pay people enough that someone /wants/ to fix it at 2am, and you can't arrange a retainer for a contractor halfway around the world to be available if needed during their worktime, and you (owner/manager) don't want to do it yourself, then it can be off until someone starts at 9am next business day. (Most products and services aren't life critical).
Nobody on any of the threads is arguing that you should not be paid more for being on call.
The quality of life lost by being on call is not insignificant. This is why being on call should always be the absolute exception and not a reliable, planned in, tool at your companies disposal.
If you regularily fail to organize your core business around business hours, maybe you should get someone who is good at planing projects and communicating these plans and put them in charge? Or maybe just close your company and search for a different career path if you are unable to plan your projects?
Besides that, the rotation is known, and planned for, long in advance. So your "long-awaited weekend trip" shouldn't have been scheduled during your on-call shift in the first place.
However, there are certainly signals that a company respects a developers' time. Things like...was there effort put in the product around things that might minimize developer on-call time, like having more user self-serve options, or spending more time on testing, or having a proper feedback loop for recurring issues to be permanently squashes in a timely fashion, etc etc. Are managers regularly a part of these off-hour on-call issues so they're affected too, thus motivated to resolve it? Or does the company just abusively unload all of the burden on the developer and force them to use up their own personal time to maintain a product? Far too often it's the latter, and I think it should be clear to all that this is abusive practice and not "normal" or "standard" nor should any of us consider it that way.
I do bring my computer with me when traveling out of state when on call but generally don’t worry about e.g. going for a few hours hike or whatever even if I won’t have the laptop with me. If it were stricter it would definitely be more annoying…
My team has an "oncall" shift like this where we assign a person to be responsible for triaging bugs and other non-urgent alerts just so we don't all stare at it and wait for somebody else (or inevitably me, the lead) to handle it but they are explicitly told that this responsibility does not extend beyond working hours.
My hours are 9-5, after that is entirely me time and if you expect me to br contact able and in s state of contribute you're going to be disappointed.
No amount of money is worth losing that me time.
which also means that this should be time that is paid on some hourly rate for the lower quality of live during the on-call time.
Also, there should be a rotation, with enough people to make this bearable.
There should be no rotation; there should be no on-call in the first place. If hiring staff to work normal hours is too expensive for the company, then the company has bigger problems than a service going down.
And having at least one daytime rotation agree that "well, you're expected to be on-call during business hours on Saturday" isn't nearly as crazy as "you can't relax at any time during the weekend".
There are plenty of people who would be happy with that and plenty more who would accept that arrangement if it came with extra compensation.
Instead, particularly at big companies, people get hired with some generic comp. package and then it's luck of the draw whether you have an on-call rotation or not, and if you do whether it's onerous or not.
You can plan around it, assuming a reasonable oncall schedule. If its every other week, yeah screw that. If its a week out of 4, you can plan it and still have it better than virtually every other highly paid profession there is, and many low pay ones.
> My hours are 9-5, after that is entirely me time
Then maybe an hourly job is better than a salaried one... Of course, if you're not paid the part, it's not worth it. But if you're paid 150, 200k or more? I hope you're not expecting 9-5.
Speak for yourself, but I would never accept a $150k a year gig that expected me to be on-call, say, 1 week out of every 6. $150k is absolutely a "9-5 and stop thinking about work when you get home" kind of salary, if we're talking California standards
Nevertheless, I'm with the parent. This practice is abusive. When you're not working, you're not working. Being on call is working. If a company wants you to work longer hours they should make it explicit, pay for it, and comply with the local employment laws. This sort of argument that the business has no merit if they were to hire more people or pay can be applied to many exploitative situations. Most of the companies doing this are drowning in profits.
I would say the default assumption for most software engineers should be they are not on call and the company should try to structure itself around that assumption. The problem starts it's just considered the norm that software engineers are expected to work 24/7.
An average company with engineers who don't hate their job, staffed adequately, it should be a stars align kind of scenario where you can't get someone on a weekend. It's a simple matter of probability, and if that probability looks grim, either A. you're understaffed, or B, your staff is underengaged.
I have fixed customer outages on my phone out having a good time more than once. Usually with another engineer happy to help. Not just at one company. If you need an on call rotation, your culture sucks. I'm not advocating that people live to work, I'm advocating that people work at places where people enjoy working with their peers and their product enough that an occasional blip isn't a big deal. With the right infrastructure and the right people you should have this.
Congratulations, you've made you entire team on call year round.
I don't read work chats outside of core working hours if I'm not currently on-call. In general, most people shouldn't. Doing so is awful for your stress levels and work-life balance. I did it for more than a decade, burning out twice in the meantime, and things have been much better since I stopped.
The goal is not to need on call, because you BOTH have a stable system that requires rare off hours interventions, AND you have a team that owns that and is bothered rarely enough to not need to have a formal rotation.
On call systems with stable systems in my experience tend to be a real problem because people lose the on call urgency anyway.
TL;DR - the problem is your system, not being on call or not.
On the other hand if it’s that less crucial for your company, 2-3 times a year without much monetary impact to warrant dedicated staff, I’m sure it can wait until necessary working morning just fine.
It’s not whether I’m being utilised or not, it’s that I’m being required to project my availability!
Do you have to be always available or can you go somewhere without connection?
I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
> There is no situation where it’s ok to contact someone to do work outside of their work hours.
It sounds like you should find a company that agrees with this opinion. I don’t agree.
And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.
There is nothing immoral about deciding you’re happy.
It is when it sends a signal (about the job, market, etc) that also affects others.
All kinds of scumbags have their "personal truth" that justifies their actions too...
That's the problem with on-call: it somehow took on moral undertones. And as a result it does not feel safe to speak out against it.
If on-call was widely seen as a negative (that a few people like because it gives them a sense of importance, more power to them) then there would be far fewer companies pushing for it as the default. As it stands, most people suffer silently for lack of an alternative. And the first step towards change is to make it OK to publicly say that on-call's a negative, a health hazard, and other options exist (though they may cost more).
More generally, free markets are doing absurd things all the time (see Matt Levine) which makes it hard to extend moral reasoning very far without getting the equivalent of divide-by-zero errors. So I don't think we should be all that concerned about how it affects the job market in general when negotiating with an employer. You know what you want better than you know what anyone else wants. Ask to be Paid More (tm) if that's what you want, but if you don't want to, you don't have to and people saying there is some kind of moral imperative to try to become even more wealthy at a faster rate can be ignored.
I personally think that a much healthier alternative would be to look for a position where one can maintain their physical and mental health, and give back extra income to worthy causes. There are many organizations much more worthy of my time and money than a bunch of managers who are too cheap to hire people for a follow-the-sun support org. But I'll accept a job with on-call if I think the rest of the deal outweighs the negatives.
It's still fair to call out on-call as a negative though. It's something to be aware of when accepting a job and for companies to keep in mind when recruiting.
If those all sound awful, by all means, do something else. But they're tradeoffs that people accept for their chosen job and compensation. My first job I would regularly be in shipyards and offshore for weeks at a time supervising some job. There was a modest allowance after some length of time but it paid better--and was almost certainly more interesting--than some routine office engineering job.
It's not necessarily some kind of inferiority complex or anything else, it's about the free market.
It is a painful slap in the face when the market doesn't price what you have to offer as high a you like. I know a friend who was convinced that her calling was to be an artist, but always had a hard time selling her works, and often complained about the unfairness of life. I guess she was "bound" by monetary considerations by not being able to make ends meet as an artist. Eventually she gave up and became a hospital lab tech and is well compensated. So the market was telling her that her skills as a hospital lab tech were much more valuable to society than her skills as an artist, even if her own preferences were otherwise.
At the same time, some other artist can buy an entire oceanside condo for one painting, because the market does value their output very highly.
That's all that we're talking about here. It is "free" but that doesn't mean that you'll be able to get whatever you want in exchange for your own output.
It's not like on call is a shock or something, it's not "whatever they throw at me", it's just... part of the job. My sister is a dentist, she has to be on call sometimes, it's not a shocker or something you don't know when you're signing up for the job.
You're implying that people aren't negotiating, but that's baseless. Software engineers are highly compensated because of these expectations, and we all negotiate accordingly.
This is quite a sweeping statement to make. My first engineering job salary was non-negotiable. I was told to either accept it, or they 'rapidly' move to another candidate.
But I don't think it's unfair to say, especially in the US, that software engineers are highly compensated.
Yes, and in many of them there are call-out fees and overtime. Programmers and sysadmins have convinced themselves that, as "professionals" they are not aligned with traditional working-class constructs like this.
Why is that any better than just getting paid overall more? Lots of companies also have internal policies like "if you get called in on a weekend take a long weekend next week" etc in my experience.
2. I doubt this is true. I've worked (and do work) for fortune 500 companies and have never ever heard of a CEO being paged for some sort of emergency. Presumably there can be some sort of crisis where they'll have to be involved within some reasonable time (extremely rare) but that's not exactly the same thing. That's why they have people working for them. That's not to say that some CEOs (especially for smaller companies) don't work very hard.
"presumably there can be some sort of crisis [...] but that's not exactly the same thing." - I'd say that it's exactly the same thing; on-call engineers are (or should be) just on an escalation path for crisis/emergency events; top managers are higher up in the escalation chain (if it's not a routine thing that others can easily resolve) but they're on escalation chain for every aspect of a large company. Of course, those events need to be rare - for example, as the original article states, up to 2-3 actual calls per year for a person being on-call; we should probably make a serious distinction between "on-call for emergencies" (which should be rare) and "routine out-of-hours support" (which might get triggered every week or even more frequently, obviously not an extraordinary event but a standard business process), which is something quite different and should have different solutions than emergency escalation.
Also, in my experience, the people who aren't getting woken up at 3am (aka. your boss) don't value the time and effort needed to stabilize these systems. So they want you working on the new shiny product feature that will get them promoted. Fixing the crappy data pipeline that shouldn't alert every other night isn't a priority when you aren't the one being woken up.
Well, it should be a priority, but I also don't work at that previous job for this exact reason. I took a nice pay increase and have never had to be on-call. Don't settle.
EDIT: Original 2000 hours comes from 40 hours a week * 50 weeks. Like I said, back of the napkin math here.
There's the rub. The issue isn't being on-call, it's not prioritizing making sure the system is robust so that on-call is boring.
In my last job there was no formal on-call, but if shit was going bad I'd be expected to resolve it, or track down the right resources to do so. In my current there is a formal on-call rotation. In my 15 years at the previous job I probably got called out of bed 3-4 times (and due to my roll I was the first call, if anyone got woken up, I did, and then had to wake anyone else needed). In my 7 months in the new job it hasn't happened yet.
When everything is on fire, it feels obvious to me that asking your $x00k employee to put it out isn't unreasonable. What is unreasonable is making that the plan instead of having robust fire-prevention systems and making that the exception.
Another way of thinking about this is that you've invested 10 dollars of your paycheck into your career. I worked way more than 50 hours a week when I started my career. Subsequently, I was able to drop out of school early (by 2 years, saving 10s of thousands of dollars), get a full time job, increase my compensation by 50% within 2 years, and then by 200% the following year when I changed companies.
By your math I'm sure I was getting paid 1/3rd of my actual compensation. But that has more than paid off in terms of the investment.
Two things.
1) I didn't just put in on-call overtime. I put in "free" hours in general.
2) I'm not really making a big assumption. Unsurprisingly working a lot of hours gave a very positive impression of me and I was able to easily justify raises and promotions when I asked.
Calling on-call hazing or bitch work is stupid, I'm not engaging with this.
Why would they pay for something they can get for free?
There is a world of difference between being called twice per year, and twice per week.
...and it's fine for that to be priced in as long as those expectations are as clear as the salary when I take the job.
Or you priced yourself down.
I agree that on-call is necessary in certain professions (e.g, doctors). I also agree that if an employee is willing to do on-call and they are compensated accordingly, then the practice is still ethical.
However, to call someone to do work outside work hours is unreasonable. On-call is considered work time, so I am expecting to be contacted during that time. However, if I'm not on-call, then it is not time for me to work, and I shouldn't be contacted by my company and feel pressured to answer the call.
Likewise, lots of engineers in tech are compensated in equity so you have a stake in the business. in that case the lines on what constitutes personal time are less defined. I don’t see it as a bad thing if I’m on vacation and need to grab a laptop and help on an incident. I’ll take an extra day of vacation later or sleep in the next day. Life doesn’t have to be so inflexible.
They said on call is ethical if an employee is willing and compensated accordingly. Additional paid time off is compensation.
Okay, now imagine that I have a baby. I offer you a .0001% ownership in the baby, and you have to get (potentially, but often actually) woken up for 1 week out of every 6.
Does that sound like an arrangement that a rational, healthy, self actualized person would agree to? Of course not. You're getting all of the downside and none of the upside. In fact in many cases the "leadership" team themselves aren't even on call!
It might be priced into your compensation but that's the exception. If you are paid "market rate" and on call is "priced in" then you're underselling your services by a large margin.
I feel like young people are more exploitable and don't understand what they're sacrificing by being available all the time. I was the same way at the beginning of my career. Fortunately I had a good team, but there were definitely times where my sleep and health suffered from being on-call because I didn't know how to say no.
Something is a little fucked with tech for doing it for free.
Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...)
I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
My brother makes about what I do, but he is an MBA and a plant manager with background in logistics.
Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop.
That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.
Instead we have self-indulgent junk like that. Wonder what my uncle would say, he was chief engineer for the gas company back in the day.
When these people begin to doubt themselves and their line of work, they’re told they have “impostor syndrome” and everything is fine.
The reason the industry does all of this is because it discourages their workers from forming unions. They trick their workers into thinking unions will slow them down.
There will come a day when the industry is flooded with enough saps that their high salary is pulled out from under them and they’ll understand the reality of the situation. They’ll be left working for pennies and at odd hours of the day.
Covid remote jobs are incapable of lowering USA salaries despite the fears of outsourcing. Your fears of people thinking they are better than they actually are are wrong - and so is the implication that "we will get what we deserve" - well, unless what we deserve is another doubling of compensation for switching jobs every 2 years...
Management should be looking to reduce callouts to zero over time.
This lacks perspective.
Imagine that your system runs banking, healthcare, telecom, etc.
Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc.
I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up.
Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people.
That's a fair way to make it work imo.
If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.
There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?
On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
1: https://firefighternow.com/firefighter-shift-schedules-and-w...
The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article).
Disclosure: I'm currently on call.
Does this actually exist? It's like an... anti-holiday. Insane.
Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22.
If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year.
Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it.
Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.
- If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees).
- While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.
Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.)
Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’
Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).
This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.
It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck.
Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.
These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done.
So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties).
It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did.
But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium.
Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by).
But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call).
Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.
Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing.
How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.
A few years back I took a job with no mention of on call and then was told there would be a fortnightly 24 hour slot, which later turned into ~weekly as team members left.
I left too.
I have no idea why they felt it reasonable to not mention this before I joined, they effectively just wasted a few months pay on me on a gamble that I had no life outside of work.
They seemed to find the idea that I don't take my laptop to the pub, or up a mountain, or on holiday, some sort of bizarre way of living. Sick system effect? Who knows.
What if you're drunk?
I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers).
You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc.
Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it.
In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).
I am always on call to secure my own house, so I have a good alarm system, cameras, locks etc which means I hopefully don't have to do much. It doesn't keep me up at night when I'm on holiday because, well, it's overwhelmingly likely that nothing will happen.
If I were on call for the neighbourhood or general area once a week, I'd probably have to physically be there patrolling it because otherwise I have liability for things which are outside of my control.
It's not as if this is some strict 1:1 relationship, we obviously had more than 1 person with knowledge of / working on a codebase.
It would have been an issue if 2-3 working on the same thing left simultaneously, but for more reasons than just on-call.
I also gravitate towards teams that work well without a lot of documentation, so that helps in case I'm really unavailable.
On the other hand, if 24/7/365 is more like a 'nice to have' option, then sure, such an arrangement will be more cost effective than more people in support; a 99% uptime SLA often can be maintained by one core person.
A competent person can dig around and fix most anything good enough in three days and that leaves you 12+ hours for other outages.
Three nines is almost 9 hours a year, but planes almost universally have wifi now, even if it's not great. If you have a bad year, with two big incidents when you're on a plane, you might not make your goal.
At four nines, it's about an hour. Good luck with that one. You're kind of screwed either way. Someone with intimitate knowledge of the system can probably handle an incident or three in time, but it's an enormous amount of work to get people up to speed for that and you can't have very many live training sessions because you don't have the outage budget. At the same time, if you page me and I'm at the cinema, I'm not going to fix it in time.
But, the biggest thing is working to make failure as graceful as possible. As much as possible, each component should be able to fail individually, without affecting the rest of the system. And, when there is an outage of a critical dependency, you need a way to push that outage status upstream and a method to manage load when it comes back up. (Those knobs should be stable and can certainly be documented for others).
Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again.
In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!
To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better.
Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).
My experience was similar in my last long-term software role. We were not explicitly on call, but we developed/deployed/supported an app which was critical to a 24×7×365 business process. So we were careful in our testing, code reviews, and deployments.
One time in 7 years I had to answer an out of hours call because of an issue in this app.
I'll take that over semimonthly on-call rotas for suites of apps my team isn't isn't responsible for.
Most jobs I've had were high trust, this was the weird exception. I didn't need a union to fix that.
I have to disagree, I can say I honestly don't hit true full stride until ~1-2 yrs of working with a complex system. A big part of this people really don't like know how to make non-complex systems, and there's next to no incentive in it with the 12-18month new project/promotion cycle. (ie, long-termism is rarely the original writer's concern)
In my experience by the time i'm 2-3 years experience with a set of systems I'm many multiples more productive and informed than someone with ~0.5-1 yrs experience on it. And I'm able to tell product/support people exactly how the system behaves in its edges, and how it would hypothetically behave if modified for new features/logic.
So, plz managers consider allowing people to become true masters of a domain too.
But there’s also something great about a company where people genuinely enjoy working for the long term and aren’t just using it as a springboard to the next opportunity.
Having a lot of employees with short time horizons leads to a lot of sacrificing long-term objectives in favor of short-term line items that sound good on a resume. That’s not to say that people can’t accomplish great things in 1-3 years, but when nobody’s time horizon extends beyond that, every goal gets altered to fit within that time frame.
Or worse, people start getting very good at starting ambitious projects, getting them to proof-of-concept phase, and then leaving because they can put it on their resume as an accomplishment. I’ve moved through some companies they were basically a graveyard of half-finished projects they nobody wanted to inherit because they all wanted to start their own thing rather than pick up the hard parts of someone else’s half finished project.
Of course, this requires the company to reward employees appropriately to make it worthwhile to stay. Many companies have figured this out, but many smaller corps underpay as long as possible and count on a steady turnover as part of their compensation strategy. Not great, IMO. Become a workplace that people want to stick with for a long time.
But... I could feel myself stagnating. I wasn't really learning anything. I left for another company and discovered that, yeah, outside of that area that I had been working on for 5 years I was almost completely incompetent. Didn't feel great. There's something to be said for switching areas every few years, it keeps your skills fresh.
Leaving this kind of comfort won’t be easy, but I know that I need to find a way.
I had thought that a switch to another company to work on new things would "fix" my burnout since it would be such a different problem space. New problems means new learning means more energy!
I couldn't have been more wrong. Leaving the comfort of something I knew to jump into the deep end of an entirely new stack just exacerbated the burnout. I've never wanted to change careers more than I have now.
So I say to you- take a break! I only had two weeks off between jobs which was just enough to start to feel relaxed. If possible, I'd suggest taking at least a month off.
For example if the expectation is that a engineering team designs, builds and maintains the machine twenty four hours a day that is responsible for the majority (if not all) of the firms profits then the majority of the rewards must flow there.
One the other hand, traditional methods for doing this have fallen by the wayside. With performance overhangs and other chicanery, what was once a lottery ticket is now a lottery ticket that someone will almost certainly try to steal from you once it becomes worth something. Instead of rewarding engineers who keep the lights on, we make them look for another job every 2 years to keep with the market.
People like to say tech is eating the world, tech is doing virtually all the killing but the hangers on seem to do the bulk of the eating.
The cost of keeping the machine running is an operational expense. There is supply and demand in the labour market. If the team keeping the machine running generates $100m of profits for the company and costs $5m in compensation, but it is possible to hire a similar team at market rates to do a comparable job for $5m , then it doesn't make much sense to pay the team more than $5m. The excess value of $95 m generated by the team's activities can be captured as profit.
It's also important not to neglect the contributions of other roles. The machine is only able to produce revenue because it is matched with customers who are a good fit for the service the machine delivers and are willing to pay for it. If there are no customers then the machine and the engineering team that support it produce no revenue. So arguably the sales team that is able to identify potential customers and help them understand if renting access to the machine could help them also provide a large amount of value to the company.
If you take the same high performing engineering team that builds and operates a valuable service in one company, and drop them into a different business context where there are simply much fewer paying customers, or the sales team isn't able to deliver, then the value the engineering team are able to generate in the new situation may be zero or negative. Maybe most of the value generated by the service isn't due to the team who built and maintain it, but the surrounding business context. Leverage is a big thing. The highest paying roles are often in situations with more leverage (in huge orgs or huge projects), not because the work is necessary more skilled or harder than work in other smaller scale situations.
If some of the salespeople and engineers believe they are getting paid disproportionately little compared to the value they create, they could consider banding together and starting their own venture where they would be the owners, not the employees. Much easier said than done, and much easier to stomach if starting in a situation where they have the financial safety net and career connections so they'll be okay if the business fails.
You're also totally normal if you respond to an incentive structure of offering senior level and above employees the perk of not having to carry a pager. Some companies use this to hire senior talent because, hey, it works.
As a side note: there's a shitload of negative solidarity in this thread. That's the attitude of: "If I must eat shit, everyone else must eat some too". I think this article (while a little flawed) makes a good pivot towards positive solidarity, and asking the question: "Is there a way to achieve this goal where everyone eats less shit?"
Backlogs represent a staffing failure. Customers are waiting 2 days for an email response? That's a staffing failure. Working harder to cover up a backlog leads to staffing issues never being addressed, and only passes that burden on to the other IT staff.
For example: a trend in recent years was static site generators. You can still find the internet littered with websites that return only errors due to SQL servers breaking long ago. But those static sites, as long as they don't depend on some other precarious tech, will keep working as long as someone's paying the bills. No security patches needed, no logs filling up disks. The servers could still go down, but it won't be your code's fault, so you shouldn't get a call.
The majority of failure risk today comes down to security and mutable state. Both of them are easier when the tech is read-only. For security, you can reduce your attack surface until no reasonable attack would work. For mutable state, you have a couple options. First, all changes to both code and data should be versioned and immutable with a working rollback mechanism. Second, never allow a change without verification, and rollback should happen immediately if verification fails. Third, do not depend on technology whose state can or does mutate regularly. Fourth, do not depend on technology that will stop operating once resource limits are exceeded. And fifth, never build something if you can buy it. (The latter has to do with investment in robustness. If you have to build an entire car, you will never have enough time or money or expertise to design a better brake system/suspension system/muffler/etc as someone who specializes in manufacturing just that one part)
Another positive solidarity take would be for everyone to stop taking jobs where 24/7 is a requirement. How do you do this? Before signing a contract, require that they put in language that says you will not be required to work outside of normal business hours. If you're a developer, have it say explicitly that you will only be required to maintain application code and not maintain servers. If they balk, don't take the job. Have a lawyer look it over. This doesn't stop the company from firing you in at-will employment states if you pull out the contract one day, so it's better to just walk if they challenge putting it in writing. Some industries just do not expect their users to be 24/7.
And a final note: if you are a developer, please don't just wing your app's design in terms of how it will work in production. Don't let the ops person take care of running it for you, and also don't try to figure it out by yourself. Put your brain together with the brain of somebody who specializes in system architecture and operations - at the start of your project, not the end. Design your app along with this other brain, to work in tandem with a very robust system architecture. And then as you build your app, ask the other brain to review it periodically and give you feedback. In terms of car analogies, this is like asking a mechanic for feedback on the parts you are designing. They've seen what fails when and why, what's painful to maintain and what isn't, what's expensive and what's not. I promise you this will result in fewer calls.
If the job posting or interviewing was upfront about the role will have on-call then the interviewee should make the decision to not apply or accept the offer if they're unable to fulfill the responsibilities.
Fairness isn't a straightforward concept, but your conception is more naive and - I'd guess - self-serving than most. How do you feel about progressive income taxes? Under a simplistic model of fairness, flat rates seem fair, but in reality the marginal ability to pay increases, and perceived pain of paying decreases, with increased income and wealth.
Now expand that concept from the monetary domain to the time domain, and consider how fair it is for people who are particularly time poor - for non-selfish reasons - to need to spend more of their time on the job. Affordances can be made for inclusiveness and they can be fair.
The other person’s situation is never a factor. They see someone getting things for free and they think they should have them for free too.
I can’t do on-call for medical reasons, nor can I work overtime. Neither of these are considered to be an essential job function for programmers, regardless of job descriptions, under ADA. A company must justify that job requirement. They can’t.
By the way, the “fairness” reason is explicitly excluded by law. It’s the entire reason ADA exists.
I worked with someone who had a sleep disorder. She would do twice as many on call shifts but only from like 8am to 8pm during the day. Someone else would take the night shift component, although that usually rotated too.
It wasn’t a bad trade because most of the on call work happened during the day. The night shifter only rarely got woken up to deal with something, so on the whole they dealt with less on call.
If people don't deserve a little slack due to personal circumstances, what happens when you find yourself in a situation where you can't pull your weight? I feel like people who think that not being able to do 110% of your job at all times makes you unqualified are in for a rude awakening when they find themselves in a similar situation.
That's dicey and depends on if you consider folks having children as them doing something they wanted for themselves, vs them doing society a favor.
If the former, it's like reducing work responsibilities of someone because they're also taking piano lessons on the side and its cutting in their free time. It's their choice and they are doing it for themselves.
If the later, then yeah, you'd have to cut them a break, in the same way we often do for folks with jury duty.
But having young children isn't, like, a health condition someone's born with.
The sleep disorder one is a bit trickier, since no one willing has sleep disorders. At the same time, it's not really your team mate's problems. I have some health issues that make me less productive at work. I don't get the roles, promotions, and compensation of the folks who can do more than I can. It sucks, but at the end of the day, companies hire you to do something. It's not a social service.
Now I’m supposed to talk to businessspeak talk and walk the promo game walk while also supposed to answer the phone at one am and stay up all night saving the business from melting down.
Yes I know I’m ranting. The industry has turned into a miserable corporate grind like finance or something, only with the added obligation of someone has to mop up to slop in the middle of the night and that someone is me.
I'll accept this responsibility so long as it comes with the corresponding authority.
Now tell me, do I really own the software?
You become highly irritable and it really brings down your mood. Eventually I couldn't deal with it after 5 years and I pretty much walked off the job. I didn't realize the toll it was taking on me and what being on call was doing to me until I was no longer on call.
The question is, what do you do about it? At some point things do have to escalate to an engineer, and I don't think it's practical to have a 24/7 engineering team staffed for every service (although having 24/7 non-eng support is very reasonable).
The best solution I've seen is having error budgets and on call compensation commensurate with required response times. Error budgets since they balance maintaining and expanding the service, compensation to respect people's time.
What have you seen that works?
I once worked for a tech company that said they had follow the sun rotation and i was dumb enough to take the job only to find out that the other countries had strict labor laws and follow the sun didn't mean the US folks would have their nights and weekends, but that the US folks were doing off hour support for the rest of the world because we don't have labor laws that protect our workers rights. I literally had to keep a seat warm from 8-5 my time to do the bulk of my work from 5-9pm and on weekends to "work around the customers" and they failed miserably at making tech work... (fin tech.. where they through money and soul sucking work at every problem)
Additionally you could still hire devs from western backgrounds. While studying abroad my cookies have led to heavy advertising from turing.com, which seems to hire overseas US devs to work remote for US companies. There's plenty of American devs in Chiang Mai working on their stealth phase startup.
edit: It seems turing.com is wasting their ad budget, because they have a 4 hour overlap with US TZ requirement. Not sure why they're advertising to IP addresses coming from Asia.
- normal rotation has 5-8 people
- on call lasts for one week
- some institutional trigger (like an SLO) exists for changing team priorities when pages happen too often
- the culture around on call is empowering, not heroic/dysfunctional
- people can still carry on with normal life while on call (just keep your laptop in a backpack and don't get drunk)
- option for next day off after picking up a page (no questions asked)
- frequent pages (more than a couple per quarter) trigger discussions about how to improve code quality or fix bug hot spots
- during on call shift you can get a break from normal sprint work (do rewarding things that add value for the team)
- there's no fear about being on call since problems are rare and everyone knows how the software systems work
- if you're stuck while debugging on call there are more senior people you can page too to get help
- work on app monitoring/debugging a little bit of the year to make debugging and triage easier when things do come up
- ops team catches the page before your team to provide context and repro
- reverting or deploying fixes isn't stressful because the deployment is stable and fully automated
I just joined a large organisation that practices this, and it's driving me nuts. Because no individual is taking ownership, nothing happens. There's no accountability, no drive to improve anything. We ask questions on Slack about "can someone deal with this problem, please?" and nothing happens. Rather than an alarm going off at 2am and the owner dealing with it (again), the alarm goes off and nothing happens, no-one fixes it. Stuff stays broken until it gets a ticket and assigned to a sprint (and even then, people self-assign to tickets, and there's no consequences to not finishing all the tickets on a sprint, so it's still relying on someone to be vaguely interested enough to pick it up and deal with it).
I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this. It's doing my head in. I think I actually prefer being "on-call" for code I built and have complete control over.
likely because the startup you co-founded would benefit hugely if you worked to fix stuff, which means you personally would benefit since you're a co-founder. In an employee relationship, this benefit doesn't flow on to the employee, and thus everyone is incentivized to do the least possible that they can get away with.
It doesn't gel with your personality, and that is why it's doing your head in.
There's lots of ways to fix this, but one of the more popular approaches is to make the on-call responsible for fixing this kind of stuff during normal working hours, and for that person to not have sprint-work during that period of time. The goal is to ensure the next person's on-call is better than yours, which leads to a continuous improvement mentality, with an end-goal of never being paged after-hours, and for all pages to be actionable.
I've been in organizations like the one you describe and the lack of initiative is often a side effect of hostility to autonomy.
There does seem to be a culture of "keeping your head down"
For years I was in this position for a trading company that had pre-market downloads at 4am my time. Bad data and my own bad code woke me up a lot ... which contributed a lot to making that code wake-me-up proof for further years.
Of course you should get paid for that time, and there's a diminishing return to such character building exercises. But eating your own dog food is worthwhile particularly in the middle of the night.
I would not accept a position that paid for on-call, even if they paid more. That just legitimizes encroaching on personal time and likely will make it happen more often. Pay me instead in equivalent PTO (or greater) so that there is a stronger forcing function to reduce pages.
As some other people have alluded here, though, in some organizations engineers seem to "escape" on-call rotations the more senior they get, which is unfortunate...
Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.
Programmers aren’t hired to do on-call. The company just decides they can skip hiring anyone for those roles by reusing “existing assets”. Same with single sources of knowledge. They don’t invest in having sufficient capacity.
Basically, companies hire people who can do multiple jobs as cheaply as possible and with has as little support as possible. Which is not sustainable.
ADA can really highlight dysfunctional systems.
For anyone saying a small company can’t afford this, ADA doesn’t apply to companies under 15 employers or that can demonstrate undo-hardship given their resources. (By the way, trying to hire employees as contractors does not work.)
But if I understand the design and code well enough to do that, why wouldn’t I also work on features and fixes in between outages? Why divide the roles when you have to choose among the same small group of people for either?
Having experienced both long-term chronic pain and long-term chronic on-call for some grumpy services, I find this comparison ... off-putting. On-call does not feel like it's even remotely close to as difficult.
Put another way, if a non-internet business wanted to operate 24/7, they would have significant expense and logistical problems (to the extent that very few do so).
If you’re a PaaS/IaaS provider, ok, you’re going to have to run 24x7. If you’re anything else, maybe you should have a serious discussion about what it’s worth to you to operate around the clock.
Of course it’s weird having web sites that are unavailable 24x7 but it’s not unheard of - some government web sites like clerks of court, inspections and GIS already are “business hours only”.
Employers have zero loyality to you as an employee and care the same amount about your career progression so in the end if you're not in for five-six (seven?) figure equity stake you're effectively a contractor / freelancer.
By being an employee you're taking unlimited liability for long tail risk of your employer that otherwise would be priced into the freelance or service contract at a very high amount.
At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary.
For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this.
SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something.
At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs.
You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing.
So here's why Google's system is better than Facebook's:
1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook;
2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough.
3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work;
4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to.
5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc.
So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert.
Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.
It boggles my mind that you suggest people lose skills by not using them often enough... these on call rotations aren't skills.
What's weird, is that we default to look at the goodness or badness of something purely on "skills" and "money".
Yes.. yes.. we've all read the SRE book and understand Googles approach to flexibility - but... it doesn't have to be that way.
Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. Sure, I made some money - but it didn't have to be at such a great cost.
> these on call rotations aren't skills.
Troubleshooting live production issues is a skill.
Triaging incidents to prioritize actions to minimize user harm is a skill.
Assessing tradeoffs of mitigation strategies is a skill.
The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc).
Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production.
> Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.
I can triag during business hours.
I can asset tradeoffs and mitigation strategies during business hours.
Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
BTW, me being anti-on call is not anti-resiliency... in fact, I don't think on-call is resilient at all with how most organizations operate since the resilience isn't adaptive capacity, but "do your day job and be on call" - no matter HOW much we talk about "your on call week is the week your on call and you focus on that" - it never happens. Never ever.
> Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
Being woken up in the middle of the night leaves me grumpy and motivates me the following work day to figure out how I can avoid getting woken up again for the same issue.
This is somewhat like dogfooding the oncall experience. If I'm trying to troubleshoot something but the logs are unhelpful or the runbook is too vague for my sleep-deprived brain to figure out, I can fix that for next time. If it takes me half an hour to answer a simple question, it's worth considering how to improve our monitoring and dashboards. If I can't follow my own documented process, how can I expect a team who didn't build these systems to do so?
As I said, I don't think oncall is for everyone. Oncall as a hidden job requirement is especially problematic, especially if your team / company doesn't expect (and/or mandate!) that your regular project work will be on the back burner during your shifts. Ideally oncall should be opt-in, but there's a balancing act with not having enough people to avoid rapid burnout. Google limits how much you can earn per oncall compensation (for SWEs, 2 weeks per quarter) -- if you're regularly exceeding that limit, then your oncall rotation isn't healthy.
There's a lot of horror stories in these comments about oncall rotations, and quite frankly I wouldn't want to join one of those either.
...
Another program Google has that's like oncall on steroids: Mission Control [0]. It's an exchange program where SWEs embed themselves in SRE teams, up to and including taking on their pager. It's a fantastic opportunity for gaining a deeper understanding / appreciation of production, but it can potentially be an even greater imposition. If you're an unattached 20-something, you might welcome the opportunity to live in Sydney for 6 months, but if you've got kids, that's not reasonable. (Of course, with the pandemic, relocation to a different country isn't really a thing at this time.)
[0] https://cloud.google.com/blog/products/gcp/adventures-in-sre...
Professionalism, natural empathy, and performance evaluations motivate people enough in my experience.
Team exchanges are great. Why would it have to be 6 months abroad though?
> . these on call rotations aren't skills.
Incident response/management is absolutely a skill.
BTW, the google model works for them... i'm being more in general since the issue was facebook vs google vs everyone else that I was replying to.
That person (or persons) is defacto on call. If you are never responsible for responding to incidents, your ability to respond to incidents will atrophy.
Like you still need people, who have the skills to do the oncall work, to man the pager during off-work hours.
Hiring a weekend person doesn't work (they don't have the skills since they're not really part of the team, + bus factor). Forcing people to work weekends doesn't work. Some sort of rotation seems the least worst option.
Which is not even all SREs
Anyways SREs are software developers as well.
The distinction SWE/SRE came up to be in times of SOX compliance; nowadays there are also some oncall-related distinctions (that SREs are more likely to be in such multiple-timezone-office teams, for instance, could be true).
Do you like having a job that gives you money to pay for all those things as well as the basic ability to, you know, live? How far do you want to take this? Is working 40 hours a week taking time away from your children, friends or hobbies? Of course it is.
But nobody owes you a living. A job is you trading your time and your skills to someone else in a way that provides that person value. There are going to be parts of that job you like and probably parts you don't.
If you really want to avoid oncall then fine, do it. That will likely limit your career opportunities because some jobs will require it or your coworkers may be rewarded (monetarily and careerwise) because they're doing something you're not. There's nothing wrong with that choice but at the same time you don't get to complain because others benefit from doing something you won't.
> but... it doesn't have to be that way.
For 24/7 production services then who exactly supports those services?
> I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore.
There's a pretty good chance that nothing you worked on will exist in 25 years. What's your point?
If you want 24/7 production services, then you need to map shifts of support. For continuous work you just run 3x8 or 2x12 hour shifts; but outside of IT industry where you need "spotty monitoring", 24 hours on / 48 hours off is a commonly used system - but it's not a 24 hour shift on top of another full time job.
Point of clarification on "losing skills" by not being oncall enough.
Google designs its SRE teams to scale sublinearly to the service, which means we're often responsible for entire portfolios, not just a single service.
It's common for individuals to be SMEs on a subset of the portfolio, as the focus of their coding projects.
Obviously the rest of the portfolio is also undergoing continuous change, and so to remain a broadly effective oncaller across the entire portfolio, you need regular production exposure to the parts you interact with less frequently.
Those are the specific bits that get rusty with disuse.
What kind of tasks are you referring to in the context of on call here? Tasks to remedy the root cause of repeated alarms paging folks?
Tasks could be filed by users of your service or external reports that get routed to your oncall or an alarm that fires because a metric moves outside of a "normal" range or whatever. Alarms ("pages") will come to you and need to be ACKed. Often they'll also generate a task. That task may require separate action or not. Many times those will just get closed or merged to existing tasks.
My team managed to get paid for oncall that we had set up with a number of false positives, click-to-fix alerts, "override if slightly above threshold", rules for not paging outside of work hours, and whatnot that makes it bearable.
That extra compensation is at 1/3 of hourly rate, so effectively it is like paying for working during weekends (1/3 * 24h = 8h) and some smaller bonus for weekdays.
Perhaps it might make up for the pandemic inflation
I'd love to see amazon and bezos get punished instead of rewarded just once for being slave drivers. Please pass federal legislation against "999" or other extremely absurd SLAs.
If you don’t like it, fucking quit? Slaves can’t do that
I would get called on days off, evenings, weekends. As late as 1am. I once brought my laptop to a Cubs game, because I knew I would get called (and I did). It was mostly because Devs broke the build, and their managers refused to have them learn how to triage that themselves because it would be extra responsibility.
I'm not against being on-call but in my opinion if I get called tonight then tomorrow morning I want to be able to start a process that prevents the problem from happening again, and I want to be able to put EVERYTHING in discussion. I don't want to hear anything about development or operations, that thing must be root caused and it must whoever can implement a fix (either me, my team or the development team, either on our side or on customer side) must drop everything they're doing and implement a correction of error.
TL;DR:
If I can't fix stuff and prevent the problem from happening again then I don't want be on call.
As someone who is oncall regularly and had close ones getting heart attacks I find this offensive. But hey, at the same time I understand the need for hyperbole to grab attention, and generally I think people get offended too easily.
Other than that my feelings about oncall are complicated. Most everyone hates it as engineers, but as customers we expect systems to be online all the time and recover quickly if they fail. Managed properly, oncall is a necessity for many products that doesn't need to be too disruptive.
Such a company is already dieing and will do once the team "on call" is used up, burned out and gone.It will be replaced in the market by something more vital, still capable to value its workforce.
MAANG all have on-call individuals even with the fact they're across all timezones and have been growing steady.
There is a world of difference between someone voluntarily accepting on-call as part of their duties in exchange for higher pay or a 4 day work week, and on-call being forced on everyone.
I’ve never heard of spot bonuses per issue, although I imagine it would create a perverse incentive to have noisy alerting that files a lot of issues. Flat bonuses per oncall shift exist (and I think they’re a great idea) but they’re relatively rare.