Hacker News Reader

Comment by luhn | Hacker News Reader

One thing I've been wondering about is how smaller organizations handle on call. Looking at the roles laid out [1] and assuming an on-call schedule of 4 weeks off, 1 week secondary, and 1 week primary, that's a team of at least 25 people.

For an organization of just a handful of engineers, how do they make on call work? A single on-call rotation would stretch the team to its limits, and it's likely that certain domains would only have a single Subject Matter Expert.

[1] https://response.pagerduty.com/before/different_roles/

View on HN

Do everything you can to eliminate the need for on-call support. It sucks, your engineers will quickly grow to hate it and will start looking for new jobs.

One of the big problems is the people making the priority software decisions sometimes forget that a software problem that often requires manual intervention at 2AM will do a lot of damage to a team if left to sit.

Certainly everything should be done to reduce the number of incidents, but you can't eliminate the need for on call.

I consider it part of my job description to get angry at anything that requires OOH escalation. Well, without drinks and pizza I guess.

Give people enough money and few enough incidents that they feel that being on call is a good deal.

I'm on a team that's under 20. Nobody complains about being on call because odds are you won't get called and you get a few hundred bucks regardless. That said we don't have to deal with week day on call because half of us are in India and when the US is sleeping they are on call (i.e. their normal workday)

Yeah, our last on call incident was like 2 weeks ago. Before that, months ago.

A long time ago we had every little thing page us. Plz no. So many false hits.

Things that can wait don't page us. It sends an email, and if we see it and have nothing else better to do, we might fix it. Otherwise it waits till work hours. Paging means something needs to be fixed now.

I have a friend who gets paged even for "low" (not really low) disk space. Here's a tip: turn up the alerts during work hours so you can resolve that junk before people on call have to deal with it.

The linked page on Alerting Principle there is good:

https://response.pagerduty.com/oncall/alerting_principles/

"We manage how we get alerted based on a simple principle, an alert is something which requires a human to perform an action. Anything else is a notification, which is something that we cannot control, and for which we cannot make any action to affect it. Notifications are useful, but they shouldn't be waking people up under any circumstance."

They also distinguish between "needs fixing right now", and "needs to be fixed asap, but can wait until office hours"...

> I have a friend who gets paged even for "low" (not really low) disk space.

I am your friend, I guess.

Fighting this battle for a couple of years now. Change is hard.

I keep saying that I couldn't care less if CPU has reached 100% in some random server, if customer metrics are unaffected. It is not an emergency. Same for disk. I don't care if it is at 80% full. Page me if it is going to run out before the weekend ends. Or in a month if it's during normal business hours.

Were it up to me, I'd tear down all these metrics and replace them with the four golden signals.

A sustainable weekly oncall rota can work with about 6-7 people, as you noted. To answer your question you collapse roles when appropriate. If you really only have 10 people then maybe it’s simply “primary” and “secondary/escalation” covering all the roles as needed. Get a little larger, to say 3 small teams and you might have the developrs in primary & secondary with all of the technical people managers/team leads/senior ICs in a dedicated “escalation management” rota that handles the “commander”, “scribe”, and “liason” roles. Dedicated support for those specialized roles only really makes sense when you get big enough that coordination becomes its own problem.

If they are doing their jobs properly: on-call should rarely wake someone up in the middle of the night. If it does, it's for an once in a blue moon issue that wasn't predicted. This is fixed and never happens again. Then in a few months something else happens, and the cycle repeats.

If they are handling it poorly: they will overwork the poor sap who gets assigned to oncall (in disfunctional companies, this is likely to be the junior-most members, who don't yet have a say on the matter). Things will be barely patched and people will quit.

In my old team we managed our on call rotation for years with five persons. One primary, no secondary, 7 days in a row. Of course we had to arrange ourselves because of vacation and sick leave but we always managed. We also had an organizational rule that forbade being on call for more than ten days in a row or on two consecutive weekends.

I was on call for years for a global lottery software and services provider in a 4 man rotation 1 week each. Took people 3-5 years to be ready to take on this responsibility. The last few years i was alone as people left with stress and i was 24/7 365. The pay was good but the day my new boss told me he didet get paid so i could escalate at nite and nobody could find a solution for months i started to look for new oppertunities. They hadent tried to prepare new people for years but managed to get a guy who was not ready to take the call but it also resulted in major incident and data loss. By the time i got out i had constant tinitus so i could barely hear on one ear from the stress so Im pretty happy Im done with it now but Sometimes miss the action ;)

the day my new boss told me he didet get paid so i could escalate at nite

Great, I want to go hourly, or if staying salaried I want to be paid as if hourly. That means I'm paid 33% for all time that I keep myself available while on-call, and paid 2x (or more?) for time spent actually responding because it's after-hours overtime.

If I'm going to be 24/7/365 with no backup and get crap about "I'm not paid for you to escalate to me at night," you're going to pay through the nose.

Thats What i told him and it quickly escalated to HR ;) and it was 30% for after hours i dident get extra for responding but was it was often at nite it would sleep in the next day.

Reply on news.ycombinator.com