Hacker News Reader

The lack of investment into decreasing on-call pain is a real factor. I work on Oracle cloud (OCI) and at least some of the orgs (VP-down) have figured out that this is something worth focusing on, and the on-call gets better and better as a result. My original team had an average of something like 50 pager-worthy (sev2) events per week until we got moved into a new org that had the right philosophy and we relentlessly drove that down because management realized that engineers made miserable by mundane ops fake-emergencies would eventually get fed up and leave, and that's not what they wanted (afaik, OCI has no such forced attrition). So we got put on a program of relentlessly tracking and categorizing the sev2 counts and committing to improving those numbers over a period of time. 25% of dev sprints were dedicated to improving ops (tools, better alarms, fixing long-backlogged bugs that led to pages), and now that team's ops are pretty easy and they are free to work on new features, which everyone prefers. I've since moved to another team whose ops had already had this optimization done, and I've never experienced a bad week of on call there.

I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues.

In my case, direct management seems interested in these issues and understand there are problems we need to fix, but ultimately the feature/product launches always make it into the sprint and the larger bug fixes never do. It's very much "actions speak louder than words".

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues.

SDE1 here. IMMV of course but on my corner of the org I've seen a bunch of team members raising concerns regarding tech debt for the L7 to shut it down as it got in the way of delivering the features he wanted to deliver.

Also the elephant in the room is how the company relies on trial by fire as a form of performance evaluation, which involves inexperienced SDEs being pushed to deliver alone chunks of major projects in spite of lack of experience or insight.

Can you share your knowledge or reading materials for how to reduce on-call load?

I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things.

If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.

I don't have any particular reading material, and my example of the on-call load at AWS that I'm referring to is probably very basic to most people.

On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis.

This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different.

It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.

There is something to be said for this approach. If the root cause is to be fixed someone needs to look at it in depth rather than running some play book procedure to recover. If you have too many problems though you're beyond the point where that helps. Let's say your software has worked flawlessly for a year, no issues, now an issue pops up, the engineers should definitely spend a lot of time understanding it, understanding why it popped up, fixing it properly and fixing the underlying process/org causes that made it pop up. It should not be "follow some playbook to recover". If issues pop up every week this is unsustainable, you're well beyond the point where stuff can actually be fixed. Automation has its own dangers, it is additional software to maintain, it has its own bugs etc. The right amount of automation makes life better for sure.

>If the root cause is to be fixed someone needs to look at it in depth

The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time.

I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked.

There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.

> the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response

Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.

This does sound pretty dysfunctional. You'd think that for something that's causing 999 on-calls getting the root cause fixed would be a priority. What I described obviously falls apart when the team has no ability to actually fix issues. Perhaps the original intent was to get those issues fixed but that somehow got lost as the org grows larger.

It is absolutely fascinating how wrong this approach is.

Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”

I'm not the parent but lemme chime in on this topic. It's pretty simple, if you don't build crappy software you won't get a heavy on-call load. You're really asking how to build great software. Build strong teams, with experienced people, follow good practices, reward quality and stability and not features or lines of code, reduce complexity, etc. etc. I've worked on software used by millions of people with a very low problem rate and then I worked on software used by hundreds of people where nothing ever works. Often in the latter the team, through lack of experience or ability, assumes that this is just the way all software is. There's plenty of examples of widely used software systems that are generally quite reliable and well built, and there's plenty of examples of stuff that's garbage, held together by duct tape, works by chance.

This is oversimplification.

One reality is more like "You are handed a system with a heavy on-call load. Make it better."

The other is "The system you built was great with 1e9 load, but now we're heading for 1e12 load. Make it better."

Sometimes throw it away and start again is the right answer

In my experience:

  1) Make a goal:  we should get paged for at most N incidents per week.  That goal should be closer to 0 than 10 IMO.
  2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately.  Example: tickets from alarms vs tickets filed by people.  Alarms having to do with external dependencies vs alarms caused by your own bugs.  Don't just make this a "it seems like we've had fewer pages lately" thing.  Real numbers.
  3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc.  There should be congratulations when they drop and a request for plans when they don't.
  4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team.
  5) dedicate time in each sprint to working on the most recent identified target from plans made in #3.

This isn't particularly complicated, and typing it out almost sounds like I'm giving you worthless common sense advice, but I think the key here is that multiple levels of your organization need to commit to making this important enough to spend time reviewing it, agree it is a priority, and put actual dedicated dev work into it.

edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)

There is a lot of SRE content about fixing on-call.

My short summary is, fixing on-call is a HUMAN problem, not a technical engineering problem.

Here’s an excellent place to start:

https://monitoring.love/articles/how-to-improve-on-call/

There is a wonderful list of talks and resources there, and those will lead you to yet more concepts and ideas to research.

> If there’s actually best-practices here that help that we’re not already doing...

Unless you have a leadership that recognizes the time sink on-call generates, all the best practices in the world are not going to help. This goes back to leadership culture. If they aren't in the trenches taking the same rotations with the on-call staff and at least simply staying up with them when the calls come in, then they will need metrics they believe to understand the deleterious effects of excessive on-call. This doesn't even begin to touch upon excessive on-call environments strongly signalling ignored technical and/or product debt that drags down new feature implementation and innovation. Leadership that truly comprehends this will allocate significant (20-25%) of time to addressing it, and push back on feature demands.

If you get to that point, then 80% of your struggle is over. Once you have leadership backing bringing the necessary time and resources to bear, reducing on-call is mostly lots of bookkeeping and documentation to identify trends and commonalities to prioritize and fix. A lot of the specific implementation details depend upon what you have available to leverage; ideally put someone in the PM role of tracking details for weekly stand-ups who is a relentless detailed note-taker and follow-upper.

> The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it.

Jeff Bezos can afford a private space program because - in part - his Amazon retail model is based on working people until they are mentally and physically broken, while paying them a pittance, discarding them, and moving onto the next group of people. He would rather spend money on crying booths and astroturf campaigns and buying newspapers than change that.

Why would he think about the expendable meat-units in AWS any differently?

Because We are knowledge workers, Not manual labourers!

/s

Related to part of what you said. I've worked in a half dozen different industries; I've worked in little companies with a half-dozen employees and globe spanning companies with tens of thousands and employees. They all think that they have special unique problems that nobody else has - 90% of it is the same problem I've seen in other companies in different industries, with different industry specific acronyms and words.

Every damn job, the same basic problems over and over, with the insistence that these problems are specific to the industry and usually to that specific company and that specific product.

Same is true in academic research (though old-timers catch it often). The common pattern is:

* approach “A” was invented in 1970 or so and didn’t work

* “B” extends “A” in multiple ways and now works

* noobs assume “B” invented “A” and treat “B” as the root of modern knowledge. “B” often has more market presence so noobs (without deep understanding) don’t see the relationship to prior attempts.

Examples:

* AlexNet/deep learning/ML in general

* MapReduce/databases/functional programming primitives

* Docker/chroot “Jail” Containers

* Bitcoin/90s coins/blockchains

Ego and ignorance are rarely a great combination :)

The Emperor's new clothes (TENC). Most people don't do history, especially the Dunning-Kruger afflicted.

Docker (Linux containers) is, like jails: awful, leaky isolation pretending to be virtualization. If you want real resource and security containment, use virtualization. Docker is insecure in so many ways; it's like using PHP to write a TLS library.

"NIH" duplication of effort with "IH" bugs.

SMH. Will they ever learn?