Hacker News Reader

Comment by AdieuToLogic | Hacker News Reader

The industry myth of "devs need to be on-call just in case prod crashes at 3am" needs to die.

First, a system failure addressed by someone awoken "at 3am" assumes said person can transition from sleep to peek analytic mode in moments. This is obviously not the case.

Second, a system failure addressed by someone awoken "at 3am" assumes said person possesses omniscient awareness of all aspects of a non-trivial system. This is obviously not the case.

Third, and finally, a system failure addressed by someone "at 3am" assumes said person can rectify the problem in real time and without any coworker collaboration nor stakeholder input.

The last is most important:

  If a resolution requires a system change, what organization
  would allow one person at 3am to make it and push to prod?

View on HN

The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

It wasn't about fixing the problem or fully understanding it, but instead making sure the system can run for the time being, get some sleep, and have a more complete triage in the morning. I've found this to work pretty well the past few years.

While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big different than the one you've experienced (which may be the more common one, I'm not sure).

So this may end up being a cultural difference between on-call situations.

> The way I was taught to be on call by a guy I worked with that was also an SRE at a large software company was to "patch the hole in the tire and get it to the service center".

That is a great philosophy to have when supporting a prod system off-hours, one which I fully subscribe.

> While I do agree with your sentiment, I'd say that the perspective I've learned about being on call is a big[sic] different than the one you've experienced (which may be the more common one, I'm not sure).

My underlying thesis is not with being on-call, but instead expecting a developer to perform their non-support duties in addition to being on-call. The worst case scenario of this is when the on-call developer is also tier one support.

If an organization wants to have developers perform SRE duties, presumably due to deeper understanding of a system, fine. Assign them to support and suspend development responsibilities during same.

Just my $0.02

In my experience we patch the hole in the tire and then get directed to patch a hole in another tire. Manager says work on the highest priority tire so now there are 2 partially broken tires but the problem is technically fixed. Repeat for 10 years and now it’s time to rebuild it “correctly”, but actually we just have two cars to patch since nobody knows what awful hacks we need to port over for “expected” behaviour

If the code is important enough that it needs to be fixed if it breaks at 03:00, someone needs to be on-call

If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.

I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

Pain from a problem needs to live where the problem can be fixed.

> If that someone isn't the dev, things breaking at 03:00 is someone else's problem from the PoV of the dev, and it will likely keep breaking.

If that's the case, then the person responding at 03:00 needs to have a serious conversation with the dev, or with whoever's assigning/prioritizing tickets to the dev.

It's not usually tickets that are needed, but a complex set of software development and reliability practices that can't be communicated through Jira.

> I've tried at several companies to get dev-teams to prioritise things causing a lot of work for the ops-team, and nothing has worked as well as disbanding the ops team and putting the devs on-call.

This is a leadership problem, not a development problem. If stakeholders prioritize resolving production issues and development teams do not address the objectives set forth, then there are bigger problems afoot.

> Pain from a problem needs to live where the problem can be fixed.

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

It's only a leadership problem if you split the responsibility for the thing so you have A: a team that builds and maintains it and B: someone else who has to wake up at night and fix it.

If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience.

> Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

It's not punitive management. Someone will have to work when the thing breaks. It can be the people who can fix the problem, or it can be someone else.

> It's only a leadership problem if you split the responsibility for the thing so you have A: a team that builds and maintains it and B: someone else who has to wake up at night and fix it.

A couple points come to mind here.

First, there is no reason for any employee to have to "wake up at night and fix it" if the organization staffs SRE's with a "follow the sun" model. Three complementary time zones are ideal, but two can suffice if one is first-shift and the other second shift.

Second and more importantly, splitting responsibility between a "dev team" and an "ops team" is problematic when the two teams do not communicate, coordinate, and assist each other in improving the system. Hence this situation qualifying as a leadership problem.

> If A and B are the same people, very little leadership is needed as they are innately motivated, and have direct experience.

A and B are different jobs, with different responsibilities, with different skills, and different success objectives. Combining the two is a recipe for burnout and/or failure.

>> Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

> It's not punitive management. Someone will have to work when the thing breaks.

The statement I quoted was:

  Pain from a problem needs to live where the problem can be fixed.

This is a punitive management policy in and of itself and unquestionably so when combined with the assertion:

  If A and B are the same people, very little leadership is
  needed as they are innately motivated ...

Two additional things to contemplate is categorizing team members as "they" as well as identifying "very little leadership" as a goal to achieve.

great points. I think over the 8 years of my SRE experience, I've probably caused a few outages after being paged at 3-4am and/or prolonged them. I've fallen asleep during one responding time (fortunately the main outage had been fixed but we went way beyond our internal SLO for backups as I had fallen asleep before running them).

That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well, and is pretty easy to staff in a large enough org

> That said, the author neglected to mention timezones or following the sun. A 12/12 hour shift or 8/8/8 (NA/EU, NA/EU/EMEA) addresses the sleep deprivation problem pretty well

I completely agree.

The "get a call at 3am" scenario, to me, is shorthand for an organization which intentionally under-staffs in order to save money. If a system has genuine 24x7 support needs, be it SLA's or inferior construction, it is incumbent upon the organization to staff accordingly.

Still and all, identifying a production issue is one thing, expecting near real-time resolution by personnel likely unfamiliar with the intricacies of every aspect of a non-trivial production system is another.

It always amazes me these places that have "24/7 support needs" then all of a sudden a bug comes in that's "not important" even though it has customer uptime impacts.

> If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?

Well yeah, if you have on call, I guess you have to trust the on-call person to make that decision. If they get called at 3am, the problem is obviously so bad that almost anything they do can only make it better, right? If it's some overlooked edge case that crashes the whole application and can be fixed with a one line change, it's probably Ok to do it a 3am. If it's more complicated, maybe wait until 7am and discuss it with others too. And, in any case, a thorough post mortem (during office hours) and checking if the 3am fix is really a proper fix for the issue is in order...

Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

The organization needs to incur financial penalties when the on-call staff have to respond at three AM. This aligns the organizations motivations with the operational motivations.

When managers see an on-call-incident line-item going up then they're more willing to tell product to take a hike.

> Devs need to be on call at three AM so that they suffer when their software fails. This is how you align developer motivation with operational motivation.

As I mentioned in a peer response:

Punitive management policies erode morale and result in retention only of those whom have no better options. See "Price's Law"[0].

0 - https://nielsbohrmann.com/prices-law/

It's interesting that you see this as being a punitive policy. Our job is producing quality systems. If our software is so bug ridden that caring for it is a punishment, then failed we've at producing quality software.

Great points.

> If a resolution requires a system change, what organization would allow one person at 3am to make it and push to prod?

The only advantage of having a micromanaging manager: if they want to review all PRs and system changes, they also must be on call every single night of the year.

Well during on-call I think usually there is a chain of wake-ups. The on-call guy tries to fix the issue but finds out he is not familiar with it so creates a channel and wakes up the next guy in the right team, and so on. The first guy usually just wakes up to "acknowledge" that someone is taking care of the issue, and writes the wrap-ups and closes the channel.

Reply on news.ycombinator.com