Tips for on Call Engineers During the Holidays | Hacker News Reader

Tips for on Call Engineers During the Holidays

transposit.com

28 pointskiyanwang28 comments

The best one is companywide code freeze well before the actual holidays (minimum 3-4 days before). That gives you 3-4 days to discover a whole slew of possibly nasty things and revert them before the week happens.

This keeps the likelihood of anything actually bad happening during the holidays way, way down.

However... it can also trigger new behaviour because programs / services run for longer than normal because deployments aren't being refreshed as frequently as normal.

This seems like it's just being contrarian for the point of it, even if it doesn't lead to any interesting conversation.

I get it, we're hackers, we like to think of edge cases and are pedantic as hell. But pedantry and edge cases aren't always useful in conversation.

This caused at least one outage in my career. Memory leak caused every single app server to crash in the space of a few hours on new years eve; regular deploys had been hiding it.

I don't think it's even that much of an edge case. We've definitely had memory leaks on all sorts of services just because they were running longer than usual without a deployment.

This is unfortunately all too common, but it can at least be solved with a simple restart - and truly fixed after the holidays, which is much better than a 4h debug fest because of a typo in some code someone wanted to quickly get in before the holidays.

We had a problem where HikariCP was slowly recycling db connections but failing to open new ones. Guess what happened when the app ran long enough the final conn in the pool closed...

I speak from experience, unfortunately :)

This. A large fraction of outages are an unintended result of a planned change, so it's wise to freeze those on weeks when no one is around and sometimes other critical times (like during Black Friday sales for a shopping site). If you do this, holiday weeks are usually super quiet.

My worst holiday oncall shift as a SRE was when a couple SWEs on a semi-related team didn't honor any of our best practices, instead rolling out an unannounced change during the production freeze with no premortem or rollback plan. (This was 12 years ago, and clearly the larger org had failed in communicating cultural expectations to everyone.)

I would go so far as to say that the vast majority of outages are caused directly by code changes. If your org has a lot of commit churn, you’ll find a few more bugs in the calm of a holiday change freeze, as hundreds of patches take their first journey past 72 hours of uptime.

exactly this, in companies that had some kind of moratorium during holidays the on-call was often the quietest time. Majority outages are triggered by code changes, even those that seemingly are unrelated.

Oh, how I don't miss being responsible for building out the holiday oncall schedule.

A few more tips:

- Lead by example. If you're responsible for building the schedule, do a shift yourself, and don't cherry pick the least bad one either.

- Remember last year's schedule: it's shitty enough to do one Christmas, it's worse to do two in a row.

- But also avoid the trap of scheduling only noobs on the worst shifts. At least the secondary should be a grizzled veteran.

- Have an escalation plan in place if shit really hits the fan. Remember the Lizard Squad attacks during Christmas 2014? Good times: https://en.wikipedia.org/wiki/Lizard_Squad

The first is the most crucial. Execs and managers sign up to take shifts.

With the numbers of Indians and Chinese in Silicon Valley is this really a problem? I'd imagine that there are plenty of engineers who couldn't care less about Christmas.

Christmas is a most secular holiday. Everyone gets together with family for Christmas. Even us Jews.

Yes, it's still a problem. New Year's Eve around midnight in particular is a shift that almost nobody wants to take.

> To reduce the pain further, consider upping on call bonuses for the entire week

Are you guys getting bonuses for being on call? I kind of signed up for it (SRE) but I'm curious if this is more common in jobs that aren't primarily concerned with reliability.

I get paid for oncall. Why would I do it otherwise?

Most places with on call shifts (that I've worked for at least - VC funded tech companies) treat it as a responsibility like any other for a salaried worker. I'm sure individuals have quietly negotiated special deals as they tend to do, but it's definitely not a common practice to pay extra for it specifically.

I guess it really varies from system to system. The one I work on has pretty strict uptime requirements so we get well compensated for it.

Of my last three employers, one had a dept/team policy of a weekly on-call bonus. The on-call person got a few hundred bucks extra (taxed, but still...), and there was a budget line item for it. It was a pretty nice token of appreciation. The other two - one had an on-call rotation no extra bonus but an informal comp time policy if you had a bad night. The other had nothing. The expectation was that the two guys on the team were on-call all the time. "Oh, we'll probably never call you."

Ironically, of the three, the one with the on-call bonus budget was the one where you almost never got called.

There are different tiers of oncall. Needing to be online and actively debugging within 5 minutes of getting paged is far more constraining than being the third line of internal escalation that gets a phone call maybe twice a year.

It's part of the salary and job description? I haven't been paid for on-call for 15 years.

Yes, but not a lot. It still adds up over time though!

Most places, if you have a half-decent manager, they just give you the time back. If you had to wake up at 3am, you get an unspoken amount of time you can claim without official PTO, on top of not having to come in at 9am. So assuming you're on call for one week, you should still only have worked 40 hours, but you can stack up that extra time to supplement a vacation or something.

I've been asked a couple of times whether or not I'd be interested in joining the on-call rotation, but I've always turned it down. I think incentive is like a couple hundred euro a month on top of your base salary. I don't actually work in ops even though I've been taking more op responsibility as time goes on, so being on-call isn't part of my job (even though the on-call people have called me at least once in the past to help sort out issues).

My last 2 clients had both on-call compensation and call-out compensation.

The former is usually a fixed amount for a shift, i.e. 5PM-9AM during the week, weekend could either be all of it or 12h shifts alternating between 2 engineers, bank holidays might also have a separate rate.

The latter is paid on top in the case of a call-out. That can either be per incident or per hour.

In case anyone is thinking this gives engineers the wrong incentive, my experience has been the opposite. Nobody minds being on-call, and not only no one is intentionally sabotaging the systems in order to get called out.

The call-out compensation is widely seen as an extra to compensate for a night of bad/interrupted sleep as opposed to being something you look forward to cashing in, and it serves as an incentive to convince managers that systems are better off being resilient on their own rather than relying on manual intervention.

Hey HN, shameless plug: I created a Slack app to help teams create on-call shift schedules: https://turnshift.app.

1. You create a schedule 2. Members get messages to book shifts 3. Notifications are sent to channels about who’s on shift

We’ve used this technique at Algolia to organize the whole-company support (40 engineers) and in general share any repetitive team task automatically.

My tool is very new so really looking for feedback. If this gets some love I’ll create a proper HN post.

Reply on news.ycombinator.com