Hacker News Reader

Comment by crispyambulance | Hacker News Reader

I think whole swathes of the SWE community are behind the times when it comes to responding to crisis.

There are models for how to handle this stuff in other industries. What comes to mind is "Tiger Teams" and "EMS services". Both of these involve the deliberate creation of special organizational structures that respond to crisis. For both of them that means contingency planning and, importantly, practice. It's largely about training for things that might happen and thus being ready when they do happen-- even if they're not exactly what was expected.

Sadly, the model for crisis management in SWE (in most places) is just throwing around the word "accountability", putting people in the hot-seat, and hoping for the best. You end up with "hero-based" crisis management. In fiction, that works out great (James Bond types always win), but in reality you just end up with stressed-out people and flaky systems.

View on HN

Back in the day (and still today, sadly) SWEs had nothing to do with how their code was run. You paid Operations Engineers to staff a NOC and build an entire practice around preventing and responding to crises. The SWEs would commit their code, push a deploy button, and go home for the day (at 5PM on a Friday), oblivious to any chaos that might ensue.

I still meet SWEs who seem to believe they have no responsibility for any failures resulting from their code if it "worked for them in dev". No responsibility to design their code to resist failures or security issues in production, nor re-design it if the current design is failure-prone. No responsibility to performance test, functional test, or check with other teams on what a change might impact at scale. No need to come up with a rollback plan or test it.

If their product makes a ton of money, upper management enables this attitude, because nobody wants to kill the golden goose or risk losing its developers. It's a question of priorities. If the business does not require SWEs to care about risk, failure, reliability, etc, then they won't.

Some companies, ones large enough to have serious SRE staffs, have 24 hour staffing of an operations center, and a War Room where people can huddle until a problem is fixed.

If they are smart, they have ways to roll back offending code.

But any company smaller than say a regional bank is going to have trouble affording this.

Now, if you using a cloud provider, you can design in rollback.

But of course there are surprises. A scenario I just thought up, when you discover on Black Friday that your order system can't handle more than 1048575 (that's 2 to the 20th minus 1, just to pick a slightly plausible number) orders in one day, and this limit is deep in your code and architecture.

The trend I've seen in big tech is that microservices architectures kind of negate a lot of this benefit though. Yes, there might be SRE's who are on-call for existential company-wide outages, and ideally they have some general knowledge about some of the biggest systems and common causes of issues. But for any issues that are due to bugs in the code or configuration of one particular service (which is usually most issues), there is a separate on-call rotation for each service. These folks are mostly just regular engineers that are primarily responsible for building features and delivering under time pressure and don't have the luxury of a lot of training & practice dealing with incidents until they happen for real.

So after a year of two of oncall at Amazon (it was long enough ago that we were oncall for the whole thing, long before SRE roles were ever invented) the biggest thing I learned was that nobody was dying and it wasn't actually fire-fighting and we shouldn't be using analogies like War Rooms and all these other War/EMS/fire analogies.

I've never worked in one of those jobs, but I've actually dragged dead bodies out of the water due to scuba diving accidents and analogizing IT outages to real life fatalities is almost sort of like how we think about holocaust/nazi comparisons and shouldn't use those casually. The industry really needs to chill out a bit about the impact, nobody is generally dying for the kinds of things we're talking about where the website is just down (obviously some IT really is life-affecting, and in finance people can be hauled off to jail by the SEC, but we're generally not talking about those industries here).

There are some processes such as accident analysis from aviation and aerospace which are useful, but could be used a bit more dispassionately. Blameless postmortems are also better than blamestorming, although I think there is a bit of a limit (at some point you may really have a bad team producing poor architecture which is constantly knocking the website over and you need to kick someone out, that isn't the purpose of an engineering postmortem though, that should happen at a management level).

I'm also weird and I don't really feel a lot of guilt (never raised religious might have something to do with it) and I just naturally understand how "shoulds" are bad and find it mildly appalling how Millennials love to throw the word "shame" around for situations where it really doesn't apply (your sportsball team shouldn't really be "shamed" if they lose). The broader population could do with a reassessment of shame and guilt and perfectionism and "shoulds".

I think some orgs are getting there, with SRE teams and red team exercises and the like. But I agree the overall industry is still deep in the 19th century, as far as managing this specific set of risks goes.

> What comes to mind is "Tiger Teams" and "EMS services"

I know EMS services of course, but what are Tiger Teams? A cursory Google search didn't turn up anything useful. Any pointers where I could learn more about them?

"Tiger Teams" are relatively common in large technical organizations (usually outside of computing).

I have seen them at a linear accelerator facility and also in large factories. I think the first time the name was used was in the NASA space program (and looking at the wikipedia link someone provided that appears to be correct).

Basically, they are teams that are put together to deal with ongoing critical cross-cutting problems in the organization. They are, by nature, cross-functional. Large orgs tend to put talent into silos. Silo'd talent can make the org vulnerable to weird problems where no one feels enough agency to act using the excuse-- "it's not my job". A Tiger Team is intended to work across silos.

It's the first time I hear about Tiger teams too. Here is a Wikipedia article : https://en.wikipedia.org/wiki/Tiger_team

Ours is a small company with about 50 employees. Whenever we encounter an especially mission-critical blocker we assemble a group of people that put their heads together to work on it. The goal is investigate all possible angles and relentlessly work on it till it is resolved.

Reply on news.ycombinator.com