If an organization thinks their systems should be available 24x7 they should staff people 24x7.
There is no situation where it’s ok to contact someone to do work outside of their work hours.
Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.
If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.
Salaries in the industry have a lot of expectations baked in, like the expectations of being able to learn and train yourself without the company having to pay for it (aside the occasional paid book). Being on call is another. That some folks can get the same pay regardless of their responsibilities or how hard their job actually is is, indeed, a problem.
Let's take this idea to it's logical extreme then. Surely you would agree that slavery is abusive. If we imagine a society where there are only slaves and slaveowners, your position would lead one to believe that slavery is not abusive, simply because it's the status quo.
This practice is abusive because it subverts the agreement that one will exchange their labor for pay within a defined set of hours (8 hours per day) and replaces it with the expectation (not agreement) that one will be "available" 24/7, but not actually "working" unless a pager goes off. This is plainly abusive, because it destroys your ability to use your free time to do things like drink alcohol, smoke marijuana, go hiking, go sailing, go for a run, etc. because you are required to be online with 15 minutes notice.
But for salaried employees, this isn't the agreement. I sometimes work four hour days. Sometimes I work odd hours. The agreement, for salaried employees, isnt about hours, it's about work getting done.
If someone consents to that, swell! I think many companies are abusive in that they don't compensate oncall work enough, but salaried oncall positions aren't inherently more abusive then having a lawyer on retainer.
My previous job I was on-call 1 week out of every 6. My first on-call week I got paged 100+ times (obv many of these alerts were firing within a minute of each other, but even accounting for that I was still being woken up multiple times per night). That week was particularly bad but I was still woken up an average of >= once per week on my on-call weeks. My team - the sole SRE team - was on-call for everything, including of course the 6 other dev teams' services.
Even worse, when I would fight to get bullshit alerts removed (the alerts that don't actually indicate a problem), I would get incredible pushback from, of all people, my own manager! (who was the manager of the SRE team since like I mentioned we were the only SRE team)
So not only were half the alerts bullshit, but I was counter incentivized to actually fix the problems causing the alerts. One great indicator of how bad things were was that everyone else on the SRE team (except the manager but he somehow always had unusually light on-call weeks) had a 5 minute delay set for pages, because the vast majority of pages would resolve within 1-3 minutes.
Oh, and by the way we weren't given any extra compensation whatsoever. Indeed I was told it was "part of the job description", which effectively meant that SRE skills were less valuable than the devs, because the devs were making identical salaries with no on-call requirement whatsoever! So a more specialized and, at least at this organization, difficult skillset, was worth less.
Anyway, I quit that job, and never looked back. I still have a friend on that exact team who is still putting up with being on-call, despite the incredible impact it has on his ability to go out and do stuff (mid 20's guy). I hear him complain about it all the time. But, like most people, he doesn't have the balls to either (a) push for internal change (which in fairness I tried and failed, although half the reason I failed was because nobody else on my team was willing to stick their neck out and push for the change with me), or (b) quit and find a new job.
So...yeah. Kind of rambling but we have a huge problem in our industry with people who either "don't have lives", or kind of do yet have so little self esteem or whatnot that they can't actually say no and protect their personal lives. In most cases on-call is simply a case of someone getting a raw deal and being too afraid to admit it to themselves.
Honestly, I'm not even sure how it's even remotely possible to draw a parallel between slavery and that. When negotiating a salary you'll have to take these things into account. Not worth it for the pay? Find another place to work. Jobs do grow on trees for people in our industry.
The question on whether it's abusive or not very obviously depends on the details. Yes, lol, I am actually willing to accept $250k per year, unlimited time off that I use upwards of 6 weeks per year of, full benefits, working from home, in the event my boss needs to call me once a year on a Saturday.
Sweeping generalizations are not helpful, and will likely hold you back in your career. You should evaluate things on their own merits.
I have bootcamp students with 6 months of software development experience making more than top developers in France, lol. The few extra Euros you get for being on call hardly seems worth it.
There's a reason French workers are almost always rioting. Low pay and very few opportunities.
No place I have ever worked has this kind of oncall policy.
Most I have seen have a response time of 1hr, note resolution, but response.
Amazon is large enough they should have people staffed 24/7 just by staggering the timezone codes at the different offices. This is how Cisco TAC works, you call them you get what ever timezone code is "day time" at that time.
So I would agree that 15min on call is abusive, luckily this is not my experience as "typical" in the industry, hell even fully staffed 24/7 call centers typically do not have a 15min response time
By working for free you’re subsidizing your organization’s unsustainable business model.
Don’t like it, quit and find a company that is moving slowly and doesn’t value ownership from engineers, someone else will fill in the vacant post. Sounds like a reasonably sustainable business model to me, especially since it has worked like this for over decades at some companies.
This has totally not been this way for decades.
Now if you're a founder, or you're being paid for this extra work or own a significant part of the business and benefit from it's success, that's a little bit different.
The lawyers at my company go home at 5 and make way more. I know a doctor who gets paid somewhere around $1500 for every day he is on call. And when they actually call him he makes even more.
Of course, you can still go to law school if you're really that jealous.
That said, there is a level of seniority at which "their problem" is "your problem." If you'd chuck the business out the door rather than getting paged a few times a year, you're not ready for that level yet. This varies by company and organization size, but fundamentally you can't (and shouldn't try to) anticipate everything up front. Sometimes you need the knowledge, judgement, or simply signing authority (literal or metaphorical) of someone specific.
I’ve long decided that unless I have the significant stock options to show for it it’s delusional. It’s what the company wants you to think: to have all the emotional investment and none of the actual ownership benefits.
If you want a cushy 9-5 job, go work for a code factory like HCL, TCS and whatnot. That way you can build crap and wash your hands after handling it to the customer.
In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
I've been in oncall roosters several times in the past. And it made freaking sure we wrote the best software we could.
I also have been on the management side of it, and the way I set it up is that whoever was the oncall engineer, if he had to work for more than 2 hours after normal working hours, he would get an additional PTO day. It actually worked quite well.
You got it backwards: if you want a crappy quality of life (or have no life and you only identify with the company that sees you as a replaceable cog), and crappy products and customer service from a company that doesn't care (to plan right, to hire accordingly, to treat its staff right) go on call and overwork, producing sleep-derived crap.
Craftsmen and artisans take their time and have boundaries.
"Feature factories", startups, and mass market crap companies forego 9-to-5 and indoctrinate naive employees that they do something important by doing so.
>In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
Yes. It's for starry-eyed naive youngsters right off the bus. The kind of people to believe they're "changing the world" by building a Facebook or Groupon.
I've been an engineer on-call for 7 years since graduating. In the meantime I handled a handful of business critical situations, developed my ability to keep cool during crisis and put my skills to the test. When I started doing it, I wasn't even paid for on-call. Now I am. The money has no bearing on me having been exploited or not. I've done it because it's cool, and when I think it's too tiring I won't do it anymore.
Am I less of an engineer, not a true craftsman (wtv that means), because of this? There's other opinions in the world, no need to be so close minded.
It's also the kind of theft that insidiously convinces you that it's really cool and that you're actually OK with it because you're learning how to 'keep cool during crisis'. You know how else your employers could have taught you that? By providing actual training during working hours, while you were being paid.
It's fine to drink of the chalice of the corporate kool aid with moderation. Still, one should never get black out drunk out of it.
We can't just chunk these complex systems over the wall and say "have fun with that" and demand more money when the solution breaks. This means we failed to properly estimate total cost of ownership when we sold them the system and now they're held hostage. This is borderline legal breach of contract.
If we told them the true sustainable cost of ownership for a follow the sun support model they likely would have never purchased the solution in the first place and we'd be out of a job.
We could have lots of annoying jobs or no jobs at all. I'll happily accept the former.
I'm not begging anyone to let me prove my technology, I'm working with management to come up with a solution that will work for the customer. Oftentimes management will forbid or otherwise prevent me from using a better, more robust technology that would have prevented more production issues because of short-sightedness. If I pointed this out and was still told to go ahead with the short-sighted solution, and it breaks in production, whose fault is that?
I'm not suggesting the 'it's your problem now, have fun' approach, I'm suggesting the 'pay overtime for overtime work' approach. This is nowhere even near to 'breach of contract', please don't throw around lawyer-like terms when you clearly don't understand what's being discussed.
That's great, but you could still have done all those things if you were being compensated appropriately for the extra time you were putting in.
It's like unpaid internships. I'm sure unpaid interns often do learn useful skills and make useful connections. But the practice is still exploitative.
Convincing them of the "coolness" of it is the most common way to exploit the naive/fresh.
The same people who do so to others, wouldn't even piss if they weren't compensated for it...
(And being exploited or not is not a personal decision. If you aren't compensated for overtime, then you are exploited. It just means you're ok with it.).
I had that too, but it was for art projects, films, music, my own programming projects or some other thing that I certainly did not do for money.
The one thing that differs here is that the people I did this for/with would cross the country if I told them I am in need.
they had a laser focus on just shipping and success theater, but little attention on actual utility of the end product. i was berated constantly for slowing things down or wanting to be more careful and do things that weren't necessarily easy but actually had potential to be useful. it's relieving to learn this is a common management deficiency.
Yeah but most of the people complaining only think they are Craftsmen but their day job is to write bad software to push ads
Let it crash. If you can't afford to pay people enough that someone /wants/ to fix it at 2am, and you can't arrange a retainer for a contractor halfway around the world to be available if needed during their worktime, and you (owner/manager) don't want to do it yourself, then it can be off until someone starts at 9am next business day. (Most products and services aren't life critical).
Nobody on any of the threads is arguing that you should not be paid more for being on call.
The quality of life lost by being on call is not insignificant. This is why being on call should always be the absolute exception and not a reliable, planned in, tool at your companies disposal.
If you regularily fail to organize your core business around business hours, maybe you should get someone who is good at planing projects and communicating these plans and put them in charge? Or maybe just close your company and search for a different career path if you are unable to plan your projects?
Besides that, the rotation is known, and planned for, long in advance. So your "long-awaited weekend trip" shouldn't have been scheduled during your on-call shift in the first place.
However, there are certainly signals that a company respects a developers' time. Things like...was there effort put in the product around things that might minimize developer on-call time, like having more user self-serve options, or spending more time on testing, or having a proper feedback loop for recurring issues to be permanently squashes in a timely fashion, etc etc. Are managers regularly a part of these off-hour on-call issues so they're affected too, thus motivated to resolve it? Or does the company just abusively unload all of the burden on the developer and force them to use up their own personal time to maintain a product? Far too often it's the latter, and I think it should be clear to all that this is abusive practice and not "normal" or "standard" nor should any of us consider it that way.
I do bring my computer with me when traveling out of state when on call but generally don’t worry about e.g. going for a few hours hike or whatever even if I won’t have the laptop with me. If it were stricter it would definitely be more annoying…
My team has an "oncall" shift like this where we assign a person to be responsible for triaging bugs and other non-urgent alerts just so we don't all stare at it and wait for somebody else (or inevitably me, the lead) to handle it but they are explicitly told that this responsibility does not extend beyond working hours.
My hours are 9-5, after that is entirely me time and if you expect me to br contact able and in s state of contribute you're going to be disappointed.
No amount of money is worth losing that me time.
which also means that this should be time that is paid on some hourly rate for the lower quality of live during the on-call time.
Also, there should be a rotation, with enough people to make this bearable.
There should be no rotation; there should be no on-call in the first place. If hiring staff to work normal hours is too expensive for the company, then the company has bigger problems than a service going down.
And having at least one daytime rotation agree that "well, you're expected to be on-call during business hours on Saturday" isn't nearly as crazy as "you can't relax at any time during the weekend".
There are plenty of people who would be happy with that and plenty more who would accept that arrangement if it came with extra compensation.
Instead, particularly at big companies, people get hired with some generic comp. package and then it's luck of the draw whether you have an on-call rotation or not, and if you do whether it's onerous or not.
You can plan around it, assuming a reasonable oncall schedule. If its every other week, yeah screw that. If its a week out of 4, you can plan it and still have it better than virtually every other highly paid profession there is, and many low pay ones.
> My hours are 9-5, after that is entirely me time
Then maybe an hourly job is better than a salaried one... Of course, if you're not paid the part, it's not worth it. But if you're paid 150, 200k or more? I hope you're not expecting 9-5.
Speak for yourself, but I would never accept a $150k a year gig that expected me to be on-call, say, 1 week out of every 6. $150k is absolutely a "9-5 and stop thinking about work when you get home" kind of salary, if we're talking California standards
Nevertheless, I'm with the parent. This practice is abusive. When you're not working, you're not working. Being on call is working. If a company wants you to work longer hours they should make it explicit, pay for it, and comply with the local employment laws. This sort of argument that the business has no merit if they were to hire more people or pay can be applied to many exploitative situations. Most of the companies doing this are drowning in profits.
I would say the default assumption for most software engineers should be they are not on call and the company should try to structure itself around that assumption. The problem starts it's just considered the norm that software engineers are expected to work 24/7.
An average company with engineers who don't hate their job, staffed adequately, it should be a stars align kind of scenario where you can't get someone on a weekend. It's a simple matter of probability, and if that probability looks grim, either A. you're understaffed, or B, your staff is underengaged.
I have fixed customer outages on my phone out having a good time more than once. Usually with another engineer happy to help. Not just at one company. If you need an on call rotation, your culture sucks. I'm not advocating that people live to work, I'm advocating that people work at places where people enjoy working with their peers and their product enough that an occasional blip isn't a big deal. With the right infrastructure and the right people you should have this.
Congratulations, you've made you entire team on call year round.
I don't read work chats outside of core working hours if I'm not currently on-call. In general, most people shouldn't. Doing so is awful for your stress levels and work-life balance. I did it for more than a decade, burning out twice in the meantime, and things have been much better since I stopped.
The goal is not to need on call, because you BOTH have a stable system that requires rare off hours interventions, AND you have a team that owns that and is bothered rarely enough to not need to have a formal rotation.
On call systems with stable systems in my experience tend to be a real problem because people lose the on call urgency anyway.
TL;DR - the problem is your system, not being on call or not.
On the other hand if it’s that less crucial for your company, 2-3 times a year without much monetary impact to warrant dedicated staff, I’m sure it can wait until necessary working morning just fine.
It’s not whether I’m being utilised or not, it’s that I’m being required to project my availability!
Do you have to be always available or can you go somewhere without connection?
I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
> There is no situation where it’s ok to contact someone to do work outside of their work hours.
It sounds like you should find a company that agrees with this opinion. I don’t agree.
And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.
There is nothing immoral about deciding you’re happy.
It is when it sends a signal (about the job, market, etc) that also affects others.
All kinds of scumbags have their "personal truth" that justifies their actions too...
That's the problem with on-call: it somehow took on moral undertones. And as a result it does not feel safe to speak out against it.
If on-call was widely seen as a negative (that a few people like because it gives them a sense of importance, more power to them) then there would be far fewer companies pushing for it as the default. As it stands, most people suffer silently for lack of an alternative. And the first step towards change is to make it OK to publicly say that on-call's a negative, a health hazard, and other options exist (though they may cost more).
More generally, free markets are doing absurd things all the time (see Matt Levine) which makes it hard to extend moral reasoning very far without getting the equivalent of divide-by-zero errors. So I don't think we should be all that concerned about how it affects the job market in general when negotiating with an employer. You know what you want better than you know what anyone else wants. Ask to be Paid More (tm) if that's what you want, but if you don't want to, you don't have to and people saying there is some kind of moral imperative to try to become even more wealthy at a faster rate can be ignored.
I personally think that a much healthier alternative would be to look for a position where one can maintain their physical and mental health, and give back extra income to worthy causes. There are many organizations much more worthy of my time and money than a bunch of managers who are too cheap to hire people for a follow-the-sun support org. But I'll accept a job with on-call if I think the rest of the deal outweighs the negatives.
It's still fair to call out on-call as a negative though. It's something to be aware of when accepting a job and for companies to keep in mind when recruiting.
If those all sound awful, by all means, do something else. But they're tradeoffs that people accept for their chosen job and compensation. My first job I would regularly be in shipyards and offshore for weeks at a time supervising some job. There was a modest allowance after some length of time but it paid better--and was almost certainly more interesting--than some routine office engineering job.
It's not necessarily some kind of inferiority complex or anything else, it's about the free market.
It is a painful slap in the face when the market doesn't price what you have to offer as high a you like. I know a friend who was convinced that her calling was to be an artist, but always had a hard time selling her works, and often complained about the unfairness of life. I guess she was "bound" by monetary considerations by not being able to make ends meet as an artist. Eventually she gave up and became a hospital lab tech and is well compensated. So the market was telling her that her skills as a hospital lab tech were much more valuable to society than her skills as an artist, even if her own preferences were otherwise.
At the same time, some other artist can buy an entire oceanside condo for one painting, because the market does value their output very highly.
That's all that we're talking about here. It is "free" but that doesn't mean that you'll be able to get whatever you want in exchange for your own output.
It's not like on call is a shock or something, it's not "whatever they throw at me", it's just... part of the job. My sister is a dentist, she has to be on call sometimes, it's not a shocker or something you don't know when you're signing up for the job.
You're implying that people aren't negotiating, but that's baseless. Software engineers are highly compensated because of these expectations, and we all negotiate accordingly.
This is quite a sweeping statement to make. My first engineering job salary was non-negotiable. I was told to either accept it, or they 'rapidly' move to another candidate.
But I don't think it's unfair to say, especially in the US, that software engineers are highly compensated.
Yes, and in many of them there are call-out fees and overtime. Programmers and sysadmins have convinced themselves that, as "professionals" they are not aligned with traditional working-class constructs like this.
Why is that any better than just getting paid overall more? Lots of companies also have internal policies like "if you get called in on a weekend take a long weekend next week" etc in my experience.
2. I doubt this is true. I've worked (and do work) for fortune 500 companies and have never ever heard of a CEO being paged for some sort of emergency. Presumably there can be some sort of crisis where they'll have to be involved within some reasonable time (extremely rare) but that's not exactly the same thing. That's why they have people working for them. That's not to say that some CEOs (especially for smaller companies) don't work very hard.
"presumably there can be some sort of crisis [...] but that's not exactly the same thing." - I'd say that it's exactly the same thing; on-call engineers are (or should be) just on an escalation path for crisis/emergency events; top managers are higher up in the escalation chain (if it's not a routine thing that others can easily resolve) but they're on escalation chain for every aspect of a large company. Of course, those events need to be rare - for example, as the original article states, up to 2-3 actual calls per year for a person being on-call; we should probably make a serious distinction between "on-call for emergencies" (which should be rare) and "routine out-of-hours support" (which might get triggered every week or even more frequently, obviously not an extraordinary event but a standard business process), which is something quite different and should have different solutions than emergency escalation.
Also, in my experience, the people who aren't getting woken up at 3am (aka. your boss) don't value the time and effort needed to stabilize these systems. So they want you working on the new shiny product feature that will get them promoted. Fixing the crappy data pipeline that shouldn't alert every other night isn't a priority when you aren't the one being woken up.
Well, it should be a priority, but I also don't work at that previous job for this exact reason. I took a nice pay increase and have never had to be on-call. Don't settle.
EDIT: Original 2000 hours comes from 40 hours a week * 50 weeks. Like I said, back of the napkin math here.
There's the rub. The issue isn't being on-call, it's not prioritizing making sure the system is robust so that on-call is boring.
In my last job there was no formal on-call, but if shit was going bad I'd be expected to resolve it, or track down the right resources to do so. In my current there is a formal on-call rotation. In my 15 years at the previous job I probably got called out of bed 3-4 times (and due to my roll I was the first call, if anyone got woken up, I did, and then had to wake anyone else needed). In my 7 months in the new job it hasn't happened yet.
When everything is on fire, it feels obvious to me that asking your $x00k employee to put it out isn't unreasonable. What is unreasonable is making that the plan instead of having robust fire-prevention systems and making that the exception.
Another way of thinking about this is that you've invested 10 dollars of your paycheck into your career. I worked way more than 50 hours a week when I started my career. Subsequently, I was able to drop out of school early (by 2 years, saving 10s of thousands of dollars), get a full time job, increase my compensation by 50% within 2 years, and then by 200% the following year when I changed companies.
By your math I'm sure I was getting paid 1/3rd of my actual compensation. But that has more than paid off in terms of the investment.
Two things.
1) I didn't just put in on-call overtime. I put in "free" hours in general.
2) I'm not really making a big assumption. Unsurprisingly working a lot of hours gave a very positive impression of me and I was able to easily justify raises and promotions when I asked.
Calling on-call hazing or bitch work is stupid, I'm not engaging with this.
Why would they pay for something they can get for free?
There is a world of difference between being called twice per year, and twice per week.
...and it's fine for that to be priced in as long as those expectations are as clear as the salary when I take the job.
Or you priced yourself down.
I agree that on-call is necessary in certain professions (e.g, doctors). I also agree that if an employee is willing to do on-call and they are compensated accordingly, then the practice is still ethical.
However, to call someone to do work outside work hours is unreasonable. On-call is considered work time, so I am expecting to be contacted during that time. However, if I'm not on-call, then it is not time for me to work, and I shouldn't be contacted by my company and feel pressured to answer the call.
Likewise, lots of engineers in tech are compensated in equity so you have a stake in the business. in that case the lines on what constitutes personal time are less defined. I don’t see it as a bad thing if I’m on vacation and need to grab a laptop and help on an incident. I’ll take an extra day of vacation later or sleep in the next day. Life doesn’t have to be so inflexible.
They said on call is ethical if an employee is willing and compensated accordingly. Additional paid time off is compensation.
Okay, now imagine that I have a baby. I offer you a .0001% ownership in the baby, and you have to get (potentially, but often actually) woken up for 1 week out of every 6.
Does that sound like an arrangement that a rational, healthy, self actualized person would agree to? Of course not. You're getting all of the downside and none of the upside. In fact in many cases the "leadership" team themselves aren't even on call!
It might be priced into your compensation but that's the exception. If you are paid "market rate" and on call is "priced in" then you're underselling your services by a large margin.
I feel like young people are more exploitable and don't understand what they're sacrificing by being available all the time. I was the same way at the beginning of my career. Fortunately I had a good team, but there were definitely times where my sleep and health suffered from being on-call because I didn't know how to say no.
There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?
On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
1: https://firefighternow.com/firefighter-shift-schedules-and-w...
It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck.
Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.
The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article).
Disclosure: I'm currently on call.
Does this actually exist? It's like an... anti-holiday. Insane.
Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22.
If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year.
Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it.
Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.
This lacks perspective.
Imagine that your system runs banking, healthcare, telecom, etc.
Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc.
I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up.
Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people.
That's a fair way to make it work imo.
These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done.
So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties).
It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did.
But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium.
Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by).
But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call).
Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.
Something is a little fucked with tech for doing it for free.
Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...)
I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
My brother makes about what I do, but he is an MBA and a plant manager with background in logistics.
Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop.
That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.
Instead we have self-indulgent junk like that. Wonder what my uncle would say, he was chief engineer for the gas company back in the day.
When these people begin to doubt themselves and their line of work, they’re told they have “impostor syndrome” and everything is fine.
The reason the industry does all of this is because it discourages their workers from forming unions. They trick their workers into thinking unions will slow them down.
There will come a day when the industry is flooded with enough saps that their high salary is pulled out from under them and they’ll understand the reality of the situation. They’ll be left working for pennies and at odd hours of the day.
Covid remote jobs are incapable of lowering USA salaries despite the fears of outsourcing. Your fears of people thinking they are better than they actually are are wrong - and so is the implication that "we will get what we deserve" - well, unless what we deserve is another doubling of compensation for switching jobs every 2 years...
Management should be looking to reduce callouts to zero over time.
If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.
This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.
Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing.
How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.
- If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees).
- While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.
Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.)
Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’
Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).