Being on Call
tyler.kim
tyler.kim
It's the loneliness. Somewhere in this vast organization connected with pipes that no one single person has put together, some element would break. Mercifully the impact ripples and you kinda know about the subsystems to track down the leak and patch it up and alert the domain owners for the next working day. You're the person they're going to call, there's no cavalry, it's you looking at incomprehensible gibberish while the rest of the world is asleep. Even the desk who has called you doesn't get involved, they're done. For me, the idea that it's just me until it's fixed (or degrades further), was the hardest thing to cope with.
I'd pair up my ideal on-call team. It'll cost more, but I'll pay it.
Incidentally pagers are still being carried in SCIFs, where there are no mobiles and sometimes no outside line.
The perspective I had was that it was a rite of passage for a systems engineer. I noticed that only the more junior engineers were on front-line duty. I believe the thinking was that by forcing junior engineers to go through pager duty, they would be much more diligent when they propose, create or approve changes down the road. Whether or not this is a perverse incentive which ultimately ruins productivity & team morale is a separate conversation...
We had a primary & backup who got paged at the same time (due to the financial implications of timing), so there wasn't a total sense of isolation. But, there was definitely that sense of helplessness when it turned out to be a problem in a totally different system (and country). Worrying about something going down definitely kept you on a tight leash with regard to WiFi and laptop accessibility. We did get a little wild with the wording on the policy. Some bars downtown had excellent WiFi, so when some of us were on call during Friday/Saturday we would just make sure we had a laptop in the car and pager on body. Definitely not the best situation to be in, but for some teams the amount of time you were on pager duty could be measured in years.
I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especially weekends. However, I actually found my greatest on-call highs were from figuring out issues in other people's stuff that was 100% unfamiliar to me, sometime at 4am. In the most memorable case of Rogue Night-Time CPU Spike I didn't even have source code access :)
The world breaks everyone.
You're not required to take the call, but you get a fairly large sum of money if you do, meaning people usually take the call despite not being required to.
Because of laws here, you're required to have 11 hours of uninterrupted "free time" after leaving work (no laws on how long a work day is), so taking a call usually means you violate that, which in turn restarts the clock on your "free time", meaning you're entitled to "sleep in" on company time the next day. Doing so of course wreaks havoc on all planning, so very few people actually take the full 11 hours off, and instead just get in "late" around 9am.
Edit: I should probably have explained that the calls happen on average once per system every 2-4 weeks, and the amount of people on call is probably limited to <5%-10% of the engineers.
Most people get in between 7am and 8am, and leave between 3pm and 4pm.
At least some of mainland Europe: Germany, France, Scandinavia, and IIRC the UK.
Europe is not a single country but every EU nations has different labor laws and working hour contracts.
I'm currently on 35, might move to 37 ( majority at my company).
Spending 9 hours at work seems awful even if your lunch break is 60 min instead of 30.
I've always liked an early start for an early finish, I was one of the very few working early though in most places.
My second software job was with a company that was just as old but much more modern (not startup-ish). the boss required the team to be in before 9am, daily team meeting was at 9.
My job now has flextime with core hours between either 10-3 or 10-2 (can't recall which) but very few people arrive after 9am. All other jobs I've worked have flextime with similar core hours, there was only one job that had a true "no rules" approach.
We have scrum inside mandatory hours. There's a fairly strict company policy that you cannot schedule _recurring_ meetings outside mandatory hours, and if you do, you cannot expect people to attend, so scrum here is 9am. Even scheduling non recurring meetings outside mandatory hours is frowned upon.
As for the schedule itself, it fits nicely with (european) family life. My kids start school at 8am, so i usually drop them off at school on my way to work, and get in around 8:30am (before COVID anyway). I get off work at 4pm, and have the entire evening with my family.
I'm aware that at least some parts of the US has (or had ?) a much more fluid line between work and free time, where people will leave work to have lunch with their family, return to work, leave to watch a soccer game (or whatever), and return to work, and leave work around 7pm.
To me that sounds much more hellish than just working 8am to 4pm :)
Also at least in Seattle it seems like every doctor’s office keeps bankers hours (M-F 9-5) so you would have to disrupt your own workday to go to an appointment, though in practice I haven’t known people to bother putting in extra work if their projects are reasonably getting done on time.
There are only two rules:
- get the job done
- be in the important meetings/calls
Unless i was in serious financial trouble, or in a learning situation, i would never accept a contract that "allows for unpaid overtime".
Most of the companies i know don't allow for "paid free time", so why should i give away my time for free to increase profits for the company ?
The exception to this is of course "job salary", but i never accept those either unless i have a say in what exactly makes up the "job" part, and i usually tell the employer that "job salary" goes both ways, so if i'm done with my job Wednesday afternoon, i'll say "enjoy your weekend" and be back next monday. That's when we usually agree on paid overtime :)
At least in the UK the law specifically forbids working more than 48 hours a week, so it's pretty common to see contracts where you opt out of this regulation. Not because the company even intends to make you work overtime, but because if you happened to work more, even entirely out of your own free will, you could still sue the company for working more than 48 hours in a week. So it's a very common thing to add in contracts here.
Just looked it up. They were actually allowed to open from 7a.m. but at least where we lived, the regular shops in the city would only open at 10a.m. Shops were allowed to open until 6:30p.m. and not sure how long they actually did.
We only had one car and lived too far from a large grocery store that would carry everything and at proper prices to walk/bike for larger volumes. So we only did them when my dad was home and could drive us (mostly - I do remember having to bike there from time to time).
But usually it meant doing it on Saturdays. Shops were allowed to open until 2p.m. on Saturdays and closed on Sundays. We definitely made use of the "long Saturday" - one Saturday per month, they were allowed to open to 6p.m.
Those who like to start their day mid morning and those who like to start their day early morning. To accommodate everyone a lot of companies use a standard "core time" of 10-2 where everyone needs to be in the office.
The earlier you start the more daylight left to enjoy at the end of the day. 9am really sounds awfully late. People who start at 9 don't enjoy their morning any more than I do. They just wake up later. And it forces them to stay longer in the evening while I'm out taking advantage of the extra day light to do outdoor activities.
I've seen some companies that clock the hours you're "in the building" and allow coming up early to leave early.
What happens in practice is that some people will come in early, start working early, and leave early. Some other people will check-in early, enjoy the office-provided amenities in the empty office, and actually start working when the rest of the team gets in the office.
Some smarter company just ask you to check-in before, say, 10 am and don't care when you leave (either early or late -- and don't require you to check-out explicitly by passing the badge) -- it's up to you and your boss/manager: as long as you meet the deadline, leave as soon or as late as you see fit.
I see a lot of threads and comments about starting at 9-10am, and I don't understand it. I'm usually eating lunch at 11am and done with work by 3:30pm. It's so nice! The few times I've had to work until 5-6pm have been awful for me. I feel like I don't really have any time to really DO anything before bed.
I am a morning person which means I really treasure my morning coffee and jazz music and reading time. I wake up at 7am and enjoy two sweet hours of a cold, dreamy morning before starting work fresh at 9am.
I am a night person which means I really treasure my nightly herbal tea and hip hop music and comic book time. I take a hot shower at 10pm before wandering into bed sleepy at 10:45pm.
None of this is rigid but I just feel like you are making super subjective arguments about a person's choice in time to work and sleep and live their life.
That said even once Covid restrictions are over forget me coming to the office every day of the week. I enjoy my life way too much to let that be taken away again. Yes I moved out here by choice and chose to take the longer commute over city life, I do realize that. But I really don't get what the problem would be with just meeting at the office like once a week or whenever 'needed' with your team instead of being forced to go and partake in all the social banter that kills concentration.
People can generally work 4-12 without much trouble or needing to recover after it other than weekends. Doing 12-8 might be an every other night kind of thing. Having a couple people handing those off might just cover it.
- the disregard for health and wellbeing of each soldier
- being required to do nonsensical things that are counterproductive and likely compromise medium-to-long term goals of the outfit you're attached to
So I agree with what you said - there is very likely a way to schedule these shifts so that these problems do not occur. The Army has little interest in this though, their requirement is that the shifts need to be covered and they currently are so there is no problem.
edit: updated comment to be country agnostic
“The “Comprehensive Crew Endurance Management Policy,” signed off Dec. 11 by Naval Surface Force Pacific and Naval Surface Force Atlantic, is the first update to the joint instruction issued just months after two 2017 fatal at-sea collisions rocked the Navy.”
Basically the crews were too tired to think clearly and crashed into things, causing damage and deaths on their ships.
https://news.usni.org/2021/01/28/latest-surface-navy-sleep-p...
My view about being a soldier, a serviceman has changed drastically during that projects. The cause is noble, and the thing they're doing is very hard. Make no mistake here.
On the other hand, they're stripped of their emotions and humanity to an extent. They become something else, an hardened being after some time.
Because, they need to do what has to be done and they shouldn't think through it, but just act.
I'm a grandson of a commander. I love soldiers, but this is how they're trained, and this is how they should be to be able to perform this duty.
So, I contend you find remarkable soldiers even in terrible groups, and that they (individuals) should not be judged on their group but on their own qualities of service.
Whether you find those parameters noble is entirely subjective and not mine to question but I do believe anyone who sincerely understands and accepts these obligations deserves respect, as an individual.
So, we should praise ISIS fighters? I am intentionally picking up example of extreme, but very real organization. Same question can be asked about SS members. They knew what they joined. Does them doing good job in that organization is really praise worthy?
The better quality their service was, the more victims they have.
You're distorting it. I was trying to point out that, they have to deform themselves to somewhat non-human states.
It's not valid only for servicemen. Firefighters are same. Police are same. Long haul truckers, cargo ship crew are same.
I think everybody making sacrifices for doing what they find worthy are eligible for some respect. They accept to deform themselves to do something they find value in.
Do not add ethnicity, politics, geography, race into this. It's about being human, and other parts are neither part of my comment nor has place in this specific thread.
> soldiers even in terrible groups, and that they (individuals) should not be judged on their group but on their own qualities of service.
Unless you was forced into that group, which was not case of SS but in some cases could be ISIS, you know what you were joining.
I don't think sacrifice itself is worth respect, if the goal of that sacrifice is destructive and wrong. If you sacrifice yourself in service of group that does genocide or tortures to get power and so on, you did genocide and tortured people so that your group have power. The ideology of "sacrifice is worth respect no matter what" empowers exactly these groups first. Believing this makes people more likely to join these.
The victims, especially those who stood against that group should be celebrated. Those who said "no" deserve respect. Those who were avoiding still deserve more respect.
Wrong is a subjective term and relative to viewing point. This is exactly what I'm avoiding here. I'm not discussing whether it's right or wrong from one particular perspective. Moreover, I'm trying to direct discussion away from war and other sensitive matters with all the power I have with my words, but you want to fixate the subject to ISIS and other terrorist organizations, that's fine.
As a citizen of a country which has to defend against terrorism in point-blank distances, I understand you, but forcing the subject to this perspective is not correct, at least in this context. Especially me, as the OP, trying to clarify it further.
My family is a victim of a genocide, but world won't acknowledge it. With your reasoning my family was on the wrong side, in a war which they weren't actively participating and was not killing anyone.
When a TV channel renames the people killing our citizens, they become heroes. They become right and we become wrong. Hah.
So, don't pull this into politics, war, genocides and other stuff. That tar pit is not a part of this conversation. It shouldn't be.
Leave it here. End of this slope is not pretty.
It is simply not true that all armies would primary defend. You make the "state run army therefore good" implication, not me. You are not leaving politics out when you claim that sacrifice of joining army is always wortht of respect. It is very political statement that prevents criticism of armies and empowers abusers in uniform.
What you are doing is trying to divorce consequences of joining those organizations and paint their members in unfairly good lights.
Sacrifice is also subjective. Some of those people like the organizations they are in.
Maybe the armies I've seen are only were defending and wasn't causing any harm, and my view is distorted on that one. I can accept that. OTOH, the core of my comment is "Accepting a hard duty and dedicating yourself to it at the level of deforming yourself is worthy of respect".
This is not a weapon/army thing. Healthcare workers are doing this for more than a year now and most people respond with "So, what?". This is just ridiculous.
> What you are doing is trying to divorce consequences of joining those organizations and paint their members in unfairly good lights.
That's a brave accusation considering the information I've disclosed in the comment you're replying to.
> Sacrifice is also subjective. Some of those people like the organizations they are in.
Liking an organization doesn't exclude sacrifices. A lot of people are sacrificing their personal and family lives because they like the corporations they work in.
The aforementioned healthcare workers also fit into this definition. This is not a job which you can do without liking/loving. It also needs some sacrifices. Both personal and interpersonal.
When I was on guard duty it was usually a 24h shift followed by a 24h break for a group of 8 or 10 privates and 2 NCOs. That means that roughly a quarter of the company is permanently detached. If you switch to 12hour shifts, you have to consider that normal operations take place during daytime. At any day you then have three groups unavailable: One on duty, one resting from the night shift, and one preparing for the night shift.
https://en.wikipedia.org/wiki/Hell_Joseon
http://www.koreatimesus.com/hazing-deaths-in-korea-disturbin...
The isn't a strategic or tactical purpose that makes this kind of situation an intentional or desired outcome. Any sufficiently large organization, and this especially holds true by militaries, are typified by the following two characteristics. First, turnover at scale necessarily prevents the rank-and-file, in aggregate, from learning and developing over time. Every year, X number of people will leave the organization for reasons the organization can't control (retirement, spouses moving, legal mandates, whatever) and they will be replaced by people who are completely unfamiliar with the organization and any lessons the people leaving may have learned. Not replacing the people leaving will leave positions unfilled, which is untenable, so they will be filled by whoever is around to fill the chair. This is inevitable and unavoidable. Second, there are only so many issues that upper management can focus on. By the time people rise up through the ranks and get into a position where they have the power to do anything, the vast majority of what it means to exercise that power is to act like a hospital at a mass casualty event - triage for the most critical issues that need your attention and trust that a solid reliance on process will prevent anything else from collapsing.
If you're looking for logic then you don't understand how these organizations work.
The logic is that everyone must be replaceable on a moments notice because they might be shot and someone else will need to pick up that job. Having roles that you need to be intimately familiar with in order to be effective is the exact opposite of what the military needs. If a job can be done by someone with minimal training while desperately sleep-deprived then that is perfect for their needs.
Exceptional workers in key (non-management) positions will take whatever they've learned with them, never to return. That leads to failed excavation projects looking for the original reasoning and business logic. Then, things need to be recreated from scratch because there is no alternative besides hiring the former worker as an expensive consultant. All because people failed to plan ahead and guarantee actual continuity, focusing exclusively on the small subset of elementary tasks. In the end, the work becomes more disorganised and product/service quality suffers because of gaps in domain knowledge, context and competence.
That myth might be relatively harmless in the primary and secondary sectors to a large degree, but is actively harmful in the tertiary, quaternary and quinary sectors. If you factor in that most of the work - both in terms of quality and volume - is done by a small minority of employees, the prevailing management dogma is costing companies lots of money to entertain the notion that worker retention is a secondary consideration. It's especially myopic because it usually doesn't take a whole lot to keep people happy in their jobs, leveraged as they are by their own obligations.
I've seen this both at large, stodgy organizations that have frequent layoffs and at smaller organizations with Frank Underwood CTOs.
Bottom performers get managed out on the firms schedule, but top performers find their way out on their own schedule. This can lead to great chaos.. people getting un-fired, throwing money at the key man risks you accidentally created, and over-correcting with too-fast over-hiring which then leads to inevitable future firings.
“By dawn, I can feel my internal organs are rotting.”
“Just gotta rinse and repeat till I’m out.”
Is the point of this entire post to invoke an existential crisis in readers?
Why would you glorify hurting yourself like this?
Wasn’t sure if it was a career based on the context.
You sound disapproving, incredulous, or perhaps mystified, but you probably have similar dynamics in your own culture that you hardly notice.
Do you not see the parallels with crunch coding, or all nighter heroics at law firms and brokerages?
Or even the obesity that inevitably results from lifestyle advertising for junk food?
Why do you think some other individuals embrace those behaviours?
I know "we do what our bosses tell us"- but honestly if it was me I would be informing my boss that this is counter-productive and will cost the company (or in this case the country and it's countrymen). Much better to have staggered shifts or even a small night crew.
I suspect you’re fairly new to the workforce. Try that for a few years, see how little effect it has but how much energy it sucks away from you, and then resign to simply doing “what the bosses want” even if it means setting piles of cash on fire on a regular basis.
Maybe it’s a European thing, but my bosses have never shied away from the ugly truth that extracting value from humans requires that the humans are able to function properly. Which includes days off, good work life balance and proper rest.
I'm commited to the company I work at any given time, from 9am to 5pm. That's all the time a company can get from me.
Bugs that get you woken up in the middle of the night are fixed in no time. Documentation that is lacking, or spaghetti code, both preventing you from quickly fixing an issue instead of spending hours on it? Fixed the next day.
If the stakes of writing bad code are low, code quality will suffer. No better fix for those low stakes than being on call.
I prefer a company with a strong tech culture: code review, thinkg long-term (marathons instead of sprints) and design systems with architecture-first on mind (instead of feature-first), automated testing, QA people, don't deploy on Fridays...
For most software systems the goal should be to abstract away the actual act of "deploying" and let the infrastructure handle that on its own.
:(
90% of developers don't work in these industries or on these types of projects. Hell, I work in healthcare, for a private company, and there is zero reason for any of us to be on call. Patient quality doesn't suffer one iota if our software is down, which it rarely is.
It really sounds like you're advocating that developers be on call to "fix" the "problem" that there's no reason for them to be on call?
- Flatulence is not "you rotting from the inside"
- I have friends that have (had) night-shifts back-to-back to a regular shift, and were on-call for for several nights a month. They never failed to answer the phone. They worked as the on-call doctor for a whole station in a clinic, not answering or not listening would have been disastrous! They admitted to be "zombies" but they were functional.
I'm not saying that this doesn't sound like hell, and I seriously am glad I never accepted a job with on-call duties, but this form of hyperbole rubs me the wrong way.
Having every other weekend cut short as you always need to be near a PC from 4pm Sunday...
The sheer terror of being woken up from a deep sleep after a long night of work just as you've finally fallen asleep...
Resigning yourself to not sleeping as you finally close out the call, full of adrenaline as you see the first glimmers of light in the sky...
Knowing the problem is the same problem as last week, last month, and that the causes are things outside your control (the 1 staffer in Asia being sick or on holiday, insufficient hardware CTO won't upgrade, bugs in orphaned upstream systems, political turf wars)...
etc
I have been in an environment for about 5 years where we Devs were on call for 7 days every month and a half-ish. This was with a business that saw traffic 24/7 with weekends and off hours being higher actually. I was called at 2am once and a couple times on a weekend and I didn't even have to open my computer. I was able to instruct the NOC in what a safe reboot procedure (that wasn't in their SOP) would be and it solved the issue. We had automated testing including performance testing with signoff required from the architect if we were too far off the baseline. This was an awesome environment to be working in and being on call was both easy money and if you ever did get a call you knew it was because something really went wrong.
Contrast that with my current environment where you couldn't pay me enough money to make me sign up for the on call rotation. Monitors that fire Pager Duty alerts multie times a day including nights. An absolute nightmare and many if not most of these self heal too.
I was at a shop with clever management that fired faster than they backfilled and left the team in such disarray that the replacements started quitting as well. Ended up from a team of 6 down to effectively me & another guy rotating support. I’m talking about getting woken up once or twice a month at 2-3am to join a live incident bridge & debug systems for 2 hours, while being asked to give an update & ETA every 15 minutes, trying not to wake your wife.. and finishing just in time to see the light in the sky. To then head into the office and deal with more fire drills all day and then get asked why feature releases were behind schedule.
Current org has auto-generating incident channels in slack which seems clever until you are on rotation and there are 5 different incident channels live at any given time with messaging popping in throughout the day as some are resolved in minutes, hours or days.
A lot of this is just management failing the marshmallow test.
Oncall is where you go in, then go home, then go in, then go home, but as you touch your back door, you are called back. You say something regrettable to the caller and then apologise. You’re so tired your almost crying and can’t get your laces done up. You drive in, can’t recall the drive and the you are due back for your day shift in 2 hours.
Contrast that with my dad. NOC calls at 2am and he has to drive I to work for half an hour. Do a 5 minute fix and drive back.
Since they knew him and that he did good work sometimes they would call him even if he wasn't actually on call. He'd ignore those calls but you still wake up... And one time they actually sent a frickin taxi to our house. So half an hour after we ignore the phone, someone rang the door bell. Taxi from company X.
He never quit that job but he signed up next union representative election and actually got elected too :)
In a 24/7/365 cargo operation for a hundred years the port chemists were on call to open their lab after hours and on weekends & holidays each time our ships came in, to operate the office no differently than when you get a vessel during the regular workday. But during regular office hours you definitely need to be there too. These ships are big and they don't always come in every day or on time very often. You get an ETA but very few things can make a vessel get there early, and many different things can delay a vessel, either case being unpredictable. For these call-outs excessive after-hours fees would be added to the nominal invoice, and good incentive pay would trickle down the food chain.
Statistically since there are only 45 office-hours in a 168-hour week, most of the vessels had always incurred call-outs.
After being in my own private practice for a while one day there was a little flood and lost everything I had worked for for 20 years.
I could recover better financially with a business than a home, and I couldn't keep both so I gave up domestic life and started staying in the lab.
Sure I was "homeless" but I had a roof over my head and I worked with what I had remaining, to survive attending to the vessels where I still had capability.
Since I was always there, I made a deal with a small contractor to get right to work whenever they needed it, charging only decent basic fees per vessel regardless of after-hours. They were expected to benefit by not passing on this arrangement to the ultimate clients who would still be charged after-hours premiums, which was most of the time.
It made survival possible and he was only revealing to clients that things were now covered 24/7 without having to wait for people to be called in. Within a few weeks all of the much bigger international operators in our ports had changed over to full 24 hr shift coverage along these lines just to compete with little old me, the homeless CEO.
How is it going now? Are you still in this business?
Actually still in the same business, it's a profession that will always be with me even if I do something completely different next.
After the flood it was clear I would have to close eventually, my lab and my life as I knew it were ruined, I just kept working in the ruins.
Closed my company seven years ago after my surviving equipment finally wore out, and went to an employer where I had already served them as a contractor since the 90's. Like any other place they needed help so I went right in and added an extra $million in capabilites the first year with just the gear they already had. Haven't stopped yet.
Of course there's staff there 24/7 now and I'm a senior scientist so it's a straight day job for me but compared to what I am known to do it's been like retirement to have all nights, weekends, and holidays off, so I've kept up 50 to 60 hr weeks the whole time, just to keep up the pace I've always had. Get more than twice as much accomplished this way, not declining performance which many people understandably find occurs to them.
Didn't stop overcoming operational problems and preparing for exponential growth, but with a few disasters hitting this company, that was not able to be deployed.
So with 12 more years progress in only the last 6 years that totals more than 80 years of progressive accomplishments in only the last 40 years.
Now with covid, economy is down and a bit of our equipment is idle too often, I'll be "retiring" and selling these enhanced solutions better than my employer can do. So I'll be an entrepreneur again since I've reached the point where their shareholders and myself will be better off with me on the outside after I've built about all the foundation that can be done for now.
Some of the people I've been working with in the labs have said I must work like Elon Musk or something.
Definitely not I said, he's much younger, I've been doing it much longer, he works like me.
But this is a guy at a desk on a night shift who seems to be mentally dead most of the time anyway, to the point where he's not able to do his duty (sleep through phones), AND he handles ammunition in this (self described) half dead state.
The military knows the risks of sleep deprivation and half awake personnel.
As far as oncall go, this is an anti-pattern. You should have metrics, alarms and tickets, and oncall should get alerted out of working hours only of high impact issues that require human intervention.
If it is about the requirement to pull all nighters or crazy schedules, some jobs do it in a regular basis (I would know, my sister is a nurse in a hospital).
This is not a programming / automated position.
https://www.bbc.co.uk/news/world-asia-56165762
(tl;dr: North Korean man swam/walked round the DMZ and wasn't spotted for hours)