Ask HN: How often are you on call?
Please share a few words on what size company you work for, and what size team you’re apart of. If you’d like to share, mention how often you’re paged or activated!
I used to work for a company that had a departmental phone and we would rotate to a different person each week to be on call. However, waking people was essentially eliminated by adding people on third shift (including me) and in India.
In both cases, I was on a team of about 5-10 in an agency or division of a couple or three hundred, within a much larger organization.
It's probably worth noting, we work with large brands and marketing agencies from all around the world and are often called upon to 'save the day' once someone's plans have fallen through. As a result, time is usually the one thing our clients don't have enough of and being able to solve their problems ASAP is what allows our clients to get back to normal operation. Given that, it's easy for me to pick up the phone at 2 am because I know that whoever is calling me is going to be having a worse time than I am.
It was kind of forced on everybody (prematurely IMHO) but respecting people's timezones and business hours was key to make it acceptable and you can take the next day off no-questions-asked if you were on call on a weekend day even if you didn't have to act on anything. It's far from ideal considering how we still react to the critical alerts and what in them usually triggers actual on-call support but I've had worse in the past so it's alright... for now.
Private cloud was originally a separate team, and we had about 5 people on a weekly rotation, where you'd do support and 24/7 on-call for a week. Now that we've merged, it's about a dozen people on the on-call rotation, though we still have a subteam of about 5 for the weekly rotation of who's paying attention to private cloud support tickets during working hours.
Thanks to some work that we did a bit after I joined, we got the off-hours paging to be much less frequent - in particular we no longer page if any hypervisor is down, only if either a whole availability zone is down, or a hypervisor hosting personal dev machines is down and it's business hours. (Otherwise, higher-level services ought to have failover between multiple VMs, and devs working odd hours can use backup machines.) Usually that second condition gets noticed by the support person (in Jira) before it escalates to the on-call person. So it's rare to have anyone actually paged, and it usually means something unexpected is on fire. So, any member of the team is as good as any other at figuring out what broke and whether it can wait until morning, which is why we were able to throw all of the combined team into the rotation without training them all in everything.
There are still improvements I want to do, but the expected number of pages rounds to zero (it looks like we've had 6 off-hours pages this year) so it's certainly manageable.
We get paged around once per 12h on average but our service is fairly young. More mature services tend to page less I think.
Last I knew it was 33.3% time credited for on call hours with 30 minute response requirement and 66.6% time credited for on call hours with 5 minute response time. And it could be taken as additional holiday or as cash.
On call: Never and never have been in 25+ years across multiple large tech companies.
Seriously considered looking into a job at Facebook where this would have been required and it was definitely one of the considerations that made that position very unappealing.
In the past I've mostly worked at small companies and have never officially been on call. I've fixed the odd out of hours issue here and there without there being a proper arrangement.
The systems I work on (trading / betting software) are such that by the time you're through testing it's very rare to hit a game breaking issue because there's a very large incentive to not like, lose the bank.
Dependencies in the critical path are kept to a minimum, etc.
Usually problems are more 'soft', like an API being turned off or changed or something.
I used to get paged dozens of times a week. Now it’s either once a week, or a few dozen times a week (usually this is a blip that can just be ignored because something went wrong and then fixed itself). Typically there’s one small actionable item every week I’m on call.
The company is 500 people, my team varies in size a lot as people quit and we try and hire replacements. It’s been as small as 3 and as big as 12.
Also, it’d be good to know if on call comes with additional compensation?
Since it is paid, we end up taking turns for the extra money. In the year I’ve worked there, haven’t been pinged even once.
My worst shift I was paged during off hours twice, but was only woken one of those times.
Is there a meaning of "work-life separation" that encompasses these professions too, or shall we say the world relies on some people not having it?
It does rely on that. Supply and demand works that way. Eventually the offered price is enough to change some minds. I'll change mine at about $3k/hour-on-call.
Being paid only if you are paged is not appropriate compensation for interrupting your personal life: having your laptop, reliable internet, cellular coverage, being available on x minutes notice.
Otherwise, companies need to hire night shifts if they want someone covering while the devs are asleep. If you want full uptime, then hire for it! Your employees shouldn't give up their sleep and weekends for a measly 0.003% of your company.
Also, I would strongly support extra pay for low-wage workers working holidays, weekends, or night shifts.
Sure, you can have a global ops team that can deal with basic system issues like "this machine ran out of disk space", but not "we're seeing a spike in 500s in our payment system because something that got deployed earlier today broke a workflow". And saying "well just roll back!" isn't a solution either. Maybe you can't roll back. Maybe that won't fix it because data is now corrupt. Maybe it wasn't even a deploy to that system that broke things, but some feature flag in some distant system. No one except the team itself can diagnose these issues appropriately.
With the former, you get some compensation, for example you may work on Saturday but have a free Monday. Depending on your priorities, it may actually be a good deal -- there are things you can do more conveniently when you have a free Monday.
With the latter, it's "when you're at work, you're at work, and when you're at home, there is still a tiny voice in your head remininding you that your boss could call you at literally any moment". This may limit the ways you spend your free time, for example you can't go to places where it would be impossible or very inconvenient to get quickly online when called. And generally, you just can't make promises even about how you spend your free time.
For a sufficiently large company, it should be possible to find a few people willing to take an unusual schedule, and then the company could be covered 24/7, with every employee able to turn off their work phone the moment they leave. It's just... more convenient for the managers to do it the other way round.
I've heard others are trying one-day rotations to make scheduling holidays fairer and more flexible for people who have commitments on certain days.
I am compensated for the oncall, but not a meaningful amount.
I am the only person on call 24/7.
Current company has 400 employees my team is around seven but we'll have to grow very fast very shortly.
only when the other guy is on vacation
I’m an SRE on a team of 4 (incl manager). The company is about 140 people. Each of my team members participates in on-call (incl manager) and is on for a week at a time. We’ve dialed things in such that we might get paged once or twice during that week and usually the pages wind up being minor/non-issues. It’s really not bad, in my opinion.