How can I get my engineers to accept being on call?
workplace.stackexchange.com
workplace.stackexchange.com
The standard in SV seems to be that on-call is required and uncompensated (some companies will dishonestly try to suggest that your salary is higher than it would be otherwise). This is gross, and we shouldn't stand for it. There are exceptions of course, but even companies I've heard of that compensate for on-call seem to fall far short of what should be expected and required.
On-call should be compensated at a base per-hour rate (outside of 9am-5pm), with an elevated rate for time spent handling an incident if the pager does go off, with a minimum payout of 1 hour of time. It should be opt-in. If not enough people want to do it to get you the coverage you want, raise the rates until your employees feel that the time is worth the money.
It's pretty messed up that OP here is throwing shade at an employee who is balking at this, wanting to put them on a PIP, plus a thinly-veiled suggestion that they're lying about having asked about on-call requirements during their interview process. Bottom line is that they're drastically changing the terms of employment without offering extra compensation, and while in many jurisdictions that's legal, it's nonetheless unethical and disgusting.
The fact is that no engineer wants to get paged. But we have to be responsible for our work and our mistakes. That said, I think it’s well understood, the dynamic of pushing rushed work. To which end, you have to construct a system where e.g. product is incentivized to align with engineering on some minimum level of reliability.
One company I worked at didn't have a formal policy of giving people time off if they got paged, but most/all managers would allow for it. That's nice, I guess, but to me still doesn't cover it. Even if the pager never goes off, the employee has made their off-hours worse (that is, not really off-hours anymore) in order to give the company value. Not compensating them for that is unethical.
Just to specifically address one bit:
> But we have to be responsible for our work and our mistakes.
Absolutely agree. But if you want me to be responsible for my work outside of regular work hours, I expect to be compensated for my time. Also consider that I'm being asked to be responsible for my teammates' work as well; while I do have some influence over the quality of their work, it's far from sufficient to guarantee their work is equivalent in quality to mine (and vice versa).
We are paid a base rate just for being on call, it's less than our normal rate and the same for anyone in the company who goes on-call. It's payment for making ourselves available, not for doing any work. If I'm up in the night, I'll want a lie in the next morning.
A company I worked at (informally, as there was no specific policy) gave people the day off if they got paged overnight and had to do work. That's certainly something. The problem with it being informal is that many employees didn't realize it'd be ok for them to take off the next day, so they didn't. And managers often wouldn't pro-actively offer it.
But the question to me is: if I have to wake up at 2am and work for 4 hours, is that "equivalent" to having the next day off? Maybe? But also: maybe not.
Then again the employer could just let the employee decide which one they want.
Probably the most severely interrupted non-working day was when I needed to take an hour between the main couse and pudding one Christmas day, when I was second line on-call and the person who was formally on-call (who doesn't celebrate) needed a second pair of eyes. While the most severely-interrupted working day was probably the one that included this outage: https://medium.com/@SkyscannerEng/how-a-couple-of-characters.... I'm quite proud of the response to that, especially that we quite carefully and deliberately stopped after one region was back up, running in a degraded state overnight in order to make sure we didn't burn out our teams by trying to do too much out of hours.
I strongly disagree if events that would trigger on call happen rarely (e.g. once per year).
Many companies have stable infrastructure but just need the extremely occasional coverage for things that rarely, if ever, happen.
I’ve seen people overreact when on call has been introduced, probably from negative experiences at prior jobs. As long as the on call events happen once in a blue moon, I don’t see why anyone would have a major problem with it.
So sure, a company fitting your description would only rarely pay out the elevated rate for someone actively handling an incident, but since an employee still has to do the exact same things when not handling an incident (regardless of how often incidents arise), the base rate needs to be there.
Like, it doesn't matter if something almost never happens because I'd be "on call" regardless- that ---is--- what it means to be on call.
I did that for many years and it didn't matter so much when my kids were little and I didn't want to, like, go to a rave off the grid or perform music over a weekend or go climbing in BFE Utah.
But I quit a job that I'd had for almost 10 years because of that fact of the work. After a while it's a really bullshit thing to do to folks.
It doesn't matter if I never get called- I still have to be available and that really is something people need to be compensated for.
You can't be on-call and go to a place you're not allowed to have your cell phone on. You can't be on-call and drink alcohol. (or other substances of various legalities) You can't be on-call and go camping in a wilderness location that may not have cell coverage. Depending on the circumstances, you may need to lug your work laptop around in your backpack.
Under such an arrangement, I control my destiny (not 100%, but close enough). If I can make the code so robust and so easy to diagnose that I'm very likely to never ever be paged, I'm ok being on call.
But if there is some psychopath product manager making engineering decisions above engineering, forcing to increase tech debt constantly, guaranteeing constant outages and thus constant on-call pages then a big fat no. I won't be on call.
Of course if everyone except a single employee IS willing to do X for $Y then the company many choose to replace them.
It's incredible how many teams have on-call for their internal service which is only used during business hours across 1-4 timezones. If your system has predictable usage and isn't global, there's very little reason to have on-call unless you have a client willing to pay $$$ or the loss of business would be catastrophic, in which case you can absolutely afford to compensate your devs for it.
Easy, you learn how to make your own sandwich. buy bulk and you amortize the cost to less than $1/sandwich
Of course, technical expertise isn't as easy as putting condiments between bread. Guess you gotta pay your labor if you're not willing to do it yourself.
Also by the same person: I work from home as a software engineer and my job is happy with my performance. But I'm putting in little effort. Am I a bad person or employee? [1]
---
I used to be a Senior Software Engineer at another company but recently I took a lower level 2 position at another company remotely. I am very good at what I do. I can usually take a well groomed agile story in any tech stack and complete it way ahead of schedule. The company always praises me for this.
But the truth is they cannot keep me busy. I know I can help out else where but my output is already so very high, especially compared to everyone else.
So instead I just sit around all day playing video games while I wait for my next work assignment. This can sometimes be days or even a week. I don't go looking for it. I have so much free time on hands that I am seriously considering getting a 2nd full time software engineering job remotely and doing this all over again.
No one is the wiser.
Does this make a bad person or employee?
[1]: https://philosophy.stackexchange.com/questions/91282/i-work-...
Hire lots and lots of staff, and work in shifts. Have enough people on each shift that even when people use PTO or fall ill, someone is always working.
Nobody ever has to be on call. Instead everyone works normal hours, but someone is always available. You’ll still have to pay extra if you want people working on holidays, though.
Anything else is devaluing yourself to the poor decisions of others. "I will work harder!".
If you really need on-call with 15min response time, design your business around it (and your finances!).
The whole blog could just stop at: > The CTO mandated [...]
The problem with every on-call article is that the best practice for a single team, a company with multiple teams, and a company with multiple continents are totally different.
Isn't that just overtime? I'd much prefer to bring a laptop with me a few weekends a year than be expected to fill a full workday with my family around the house on a weekend.
I think the only viable method for such a fast response time is that it's your work. You will have off on Friday but work on Saturday (or something like that).
If it's more like 1h response time sure, take your laptop with you to familys dinner.
I think there is certainly merit to wondering what format a business needs based on client demands, but a mentally focused white collar job doesn't need to go as extreme as a firefighter who is literally burning out their body in the line of duty.
My main point is along the line if your customer wants these nines, they have to do (pay) something. If you as employer want it, same. You as employee are just at "the end of it".
Base compensation which you receive regardless of whether there are any incidents during the week or not, plus hourly compensation (50-100% overtime + another 40% for the inconvenience)
Also, those who have opted in are quite flexible when it comes to covering for each other for a few hours - so you can still go to the movies or host a birthday party for a kid without fear of the phone suddenly buzzing.
Result? People are doing their best to get in on the roster, as the compensation feels worth it.
Try that.
Secondly, there is a cultural divide sometimes. Generally, iOS programmers aren't on call, because they can't ship anything in less than a day or two. So, you get this phenomenon where they say "I'm an expert in Apple user interfaces" (a fine thing to know about!), but everyone resents them, because they are not on call.
No, fire them without cause like a grown up and take the hit to your unemployment insurance. Don't put them through some three-ring circus act masquerading as employee development. It's undignified and disrespectful. You and the employee have a disagreement over the job description -- this not performance related. Don't try and spin it as that.
It amazes me that these B-school pricks managed to enshrine at-will employment across the country, and they still won't just fucking fire people like adults. No, instead we got one of the worst MBA-isms visited on corporate America, the PIP. Fucking cowards, the lot of them.
> on call at least once a week, including the work day and evening, once a month, including the weekend of that week.
> I should also note there is a required 15 min response time to pages. The person MUST be able to be online and fixing the problem within 15 mins of the page.
I've worked a position where I had extra duties on the weekend, so I effectively worked 12+ days in a row many times. I've also been on an on-call schedule like this with a short response time. Both of these severely impact your personal life.
You have to realize that the two most valuable things you can give engineers is their own time back and money.
This is how I would probably try to handle it: To start, it would be opt-in and not mandatory, as this type of responsibility is difficult for those with small children or other family members, such as elderly, needing care. The 15 minute response time also means you should be being paid an hourly rate (maybe even 1.5x or 2x) while being on-call, whether there's an incident or not. This is because it severely limits what you can do -- you're basically chained to your laptop and an internet connection. Anyone responding to an on-call event after hours should also be given a half or full day off the next work day. Weekend on-call should at a minimum be given a day off the following week, preferable two.
I'm generally anti-on-call, but c'mon, this is an unreasonable standard. No system is 100% perfect, never needing manual intervention to get things going again when something happens.
Any business that has a service that is supposed to be running 24/7 will likely need some form of on-call.
If a business can't or won't make those arrangements (and pay the bill for them) then maybe its management team need to recognise that fully reliable 24/7 operation is not something they can guarantee and they should plan accordingly.
My phrasing could have been more clear, I'm not saying you need a 100% perfect system and no on-call.
It's weird they're instituting on-call for everyone now, and I'm curious about the reason behind the policy change. I've heard of "on call for everyone" being used as both a response to programmers shipping unreliable code, and also as a preventative measure to encourage programmers to ship more reliable code.
Also, this "manager" sounds totally clueless...
PIPing without an actual performance problem is also constructive dismissal, and doing it because the employee is refusing a change is possibly retaliation.
Realistically, in the U.S. at least, there's very little risk that the employer will get sued, but it's still shitty and will eventually bite the employer if they get into a habit of this.
As soon as something happens, you jump to regular overtime pay so if it's on a weekend+Sunday, you're making around 200-300% pay. If it's just after others logged off, you're getting normal pay, maybe 50% extra depending on the company.
I know a lot of people who have paid off loans just by hogging all the on-call hours they can. It's not especially hard if the playbooks are well made and nothing goes horribly wrong.
So, in real life, need agile approach.
Some people just accept to be on call all time, you could consider them as your core team with appropriate benefits and value them, and respect them, really.
Second group of people could accept receive calls during their shift. This is also not bad. Value this.
And exists people, who are good workers, make great value, but could be on call very limited time or nearly no time. You should decide, with them individually, how you will work. It is really possible, some very valuable people will not be on call at all.
- $700 for any week that I was on call (whether or not there were any incidents, and more often than not there were none)
- paid my phone bill, roughly $100/mo (including data + amortized phone cost)
- free to take off the next day any time I'm paged off-hours.
There have been other times I was unofficially on-call (at small startups/companies) with no compensation, but I don't think there was enough scale/frequency for it to be a sticking point.
An average week of out-of-hours involves precicely zero pages. This is also important: when we get paged for something, we take steps to avoid being paged for the same thing again. Not just because we value our sleep, but because management values our services being up and running so we're encouraged to fix stuff that's broken.
We did have a team saying that their services were noisier than other teams', and asking if they could be paid more as a consequence. The answer was to fix their service reliability, not to incentivise people to wait until out-of-hours to declare an incident so they trigger the bonus payment.
So long as I've got mobile coverage, I'm free to get on with my life. I carry my laptop if I'm more than 30 minutes from home, but I don't expect to need to use it.
how so. It's already the incentive to work. If you require more work and more of a employee's time, you pay more money. It's not that complicated.
In the same vein, there's expected business hours, so requiring a business to operate outside of those require more money. If they need that sort of timely response off hours, they pay more money too. So the incentive should be "try to call us during business hours or wait for a response".
What I don't like is "if my service is flaky then I get more money", because I don't want my service to be flaky but I like getting money.
If you're paying me to be available, but giving them time in lieu for any time worked, then you're not getting more of my time for free. Conversely, if you pay me only when I'm called out (which I recognise isn't something you've suggested) then I get the hassle of being on call but I'm only paid if my service breaks.
I don't doubt this, but surely you can agree many people lead lives which they would not be free to live in this arrangement? Putting children to bed. Driving somewhere. Cooking. Going to a bar...
I've also dropped off the on-call rota, and encouraged others to do the same, to care for very young children or family who need care that would be compromised by responding to a page. Or vice-versa, of course: if I've already been up half the night looking after a newborn then I'm not going to be much help. My contract doesn't oblige me to be on-call, I wouldn't do it if the compensation didn't make it worthwhile.
How nice, uncondemning it is, how it really gives the possibility of the OP is truly clueless that he's in the wrong.
I used to work on call in non-software, as a blood analyzer Field Service Engineer for medtech companies, like Siemens, Roche, Beckman-Coulter, Bio-Rad, etc. I spent 5 years doing this.
In that industry and job it's standard to work on call rotations that last for 7 days through the weekend and repeat every 3-5 weeks depending on team size. Because the job is hands-on and at customer sites, the minimum response time is usually a bit more than 15 minutes.
If anyone is considering accepting an on-call rotation, don't. It takes over your life. At first you may think it's fun to swoop in and be the hero fixit person, or maybe you at least tolerate it because of high base pay or something. Eventually you grow to hate the sound that accompanies the callout text or email. By the end, it triggered a visceral physical negative reaction I later learned was a minor anxiety attack.
The thing managers seeking to add on-call don't understand (those who haven't done it themselves), is that when you're on call you can't do anything. Not only can you not drink (minor annoyance), you can't even leave the house for normal stuff like a movie (medium), or attend events like a kids graduation or recital (major).
And before you say "just swap rotations around on your team," I worked with about 10-15 guys and you'd be surprised how hard it is to swap. Everyone makes plans for things all the time without realizing it would interfere with on-call, like going to the gym. At the start of the year when the on-call rotations would be released to us, there was a mad rush to submit vacation requests to get it locked in before illness or other things required swapping on-call around, much less having to take field requests from colleagues that you may (will) have to rely on when you in turn need a swap, so you can't just ignore every request.
This manager and CTO are hilariously unprepared and uninformed, bordering on pure dreaming. A 15 minute max response time is criminally unrealistic, you can't even go to the grocery store. And once a week seems low on paper until you realize that depending on team size, you can easily wind up being forced to give up a quarter of your weekends for the year (as I had to).
I haven't worked on-call for software, so maybe it's different, but I doubt it's really too different.
After 5 years of all that lost time, the amount I'd have to be paid to go back on-call is so astronomically high no company would ever pay it. Actually, now that I have a kid, there's probably no amount I could be paid if it means missing time with them.
I still hated it and don't want to do it anymore because we had so many factors out of our control that made "false" alarms where the only thing to do was check if it was really a problem and then open a ticket upstream at a vendor...
The correct response to this IMHO is for the existing staff to collectively tell management where to go. In life you can't always get what you want if you don't have the money or make the effort it requires.
For a US exempt employee it would be fairly straightforward to push oncall expectations into a role. But in other countries with robust labor protections it might be more challenging. Ultimately the solution is money.
But doing it yourself goes a long way to convincing your team it’s necessary. I personally do this as a manager and I find it helps a lot. If you get paged a few times at 3am it really helps align your priorities with your team.
I recommend investing in real QA and dumping the move-fast-and-break-stuff attitude and probably 90% of the madness that gets bundled under the Agile and DevOps banners as an excuse for not planning or designing anything properly any more. Your Operations team who are running a 24/7 schedule (and hiring for it) anyway should be able to deal with genuine emergencies without requiring non-specialist developers to be on call routinely. The world has worked that way forever and still does outside the DevOps bubble - including numerous industries that operate 24/7 and rely on software to do so.
Anyway, lots of jobs with variable demand (doctors, vets, supers, utility workers, etc.) work on a fixed / % of salary daily rate + time-and-a-half when activated. What I agree is bullshit is TFA though, if it's not in your contract, don't do it.
Current employer is trying to have me on call for free 24/7. Not happening.
The "easy" answer is to put it in the job ad, and hire people who agree to it. Because forcing that requirement on an existing employee just isn't going to happen.
* by all means, pay me an indecent wage, enough that I can retire after a year!
I'm with you. On-call is never worth it. If a company needs 24/7 coverage, they should hire round the clock.
At the very least, make it opt-in, and keep raising the on-call pay until you have enough existing employees who will decide the trade off is worth it. At some point it maybe ends up being more economical to follow your suggestion to put out a job ad specifically for on-call people.
I did on call 24/7/365 for 23 years. I'll never do it again. My phone goes silent in the evening and doesn't come online again until I get up to come to work. That's never going to change. I'm not 20 anymore, I don't function on no sleep.
I hope you made enough money to retire comfortably. That sounds like utter hell.
Made that mistake. Glad it only took one layoff to show that that's never true in Big Business. Never again. I am a worker providing my talent to work on a product I don't own. I will take responsibilities but never "ownership". "ownership" implies that I have a stake in the product, and the product goes with me no matter what. Which I have yet to experience (maybe one day).