When Things Go Wrong | Hacker News Reader

When Things Go Wrong

eng.localytics.com

45 pointsabuggia20 comments

I disagree with

"Why didn't [Engineer X] know enough SQL to understand that query would break the database?"

being bad (While "We need to shut off his/her access." is).

Post mortems have many different levels, technology -> people -> processes -> org -> culture. It's not wrong talking about people, it's wrong to blame them though and stop with people. Asking 'Why?' only stops with company culture.

Usually people make mistakes, they don't want to break things. It doesn't help to not talk about people, but it's important to ask why the person in question acted the way (s)he did. Would others do the same? (most probably) Why? UI issues? Documentation issues? Too many concurrent warnings? etc.

Also important to look from the situation forward (incident happening), not from now backwards (hindsight).

To the SQL question one could ask about the training, reviews, ...

Author here. I agree with you that training engineers on SQL would be a great action item coming from a postmortem. I was trying to emphasize resisting shutting off access as "the fix" for the problem. We don't want to blame engineers for not having the knowledge - we want to level up their skills if that contributed to the issue. I think you're asking the right kinds of questions.

Is anyone else getting this (using uBlock with FF):

  uBlock Origin has prevented the following page from loading:
  http://eng.localytics.com/when-things-go-wrong/
  Because of the following filter
  ||localytics.com^
  Found in: Malvertising filter list by Disconnect • Basic tracking list by Disconnect

Yes, and after Googling what they do, it makes sense.

Thanks, was thinking it might have been a problem with the uBlock plugin but after Googling them myself I wasn't doubting that much anymore ;)

Blocking the actual page as well and not just scripts? Seems broken to me.

uMatrix is more granular (and allows images and css by default) but still block the whole domain. They're in the business of tracking people so it makes sense. Whitelisting eng.localytics.com while still blacklisting all other *.localytics.com is another possibility.

Not really. Why waste time blocking specific scripts when the whole point of the company is to be sleazy?

There's really never a situation where I want to download something from a site like that, because I don't trust them at all and I completely disagree with what they are doing.

It's not worth whitelisting - I can read something else.

Good for you. Dealbreaker for me.

Thanks to gorhill there is now an update to uBlock to address this.

https://twitter.com/gorhill/status/776183271210229760

Huh, guess I'm not the only one to use uBlock as their preferred adblocker. IMO, I always trust them when a blocked page pops up.

The whole blameless postmortem is kind of laughable coming from a company well known (at least in the Boston area) for a culture of abuse towards engineers and a high turnover.

Huh? Abuse? I've been working in the Boston tech community for a long time and have not heard this. Localytics as an engineering culture has always been top notch. Yes, the company has lost people, good people at that, but the team is still doing very innovative things. Ive spoken with people who are there now who still say its still a very good engineering environment.

I'm a big fan of using chat as the real-time incident log even when we're all in the same room. It's trivial to generate a timeline of events for the postmortem and there's a lot of info you can extract from the log during the postmortem process.

Yes, I think that's what has added most value for us after introducing Slack. It's extremely useful.

I'm working on some communications software myself - so one of my hobbies is writing down as many UX issues as I can find with slack/hipchat/etc etc

Have you noticed anything that you'd change? I actually like slack for the most part, but I still find some UX smells in it.

And I actually use it for incident logging too.. never really thought about it but it's one of my favorite uses for it - and it helps the client on that job see the response times and get a general overview.

> The chat history is a convenient record of what happened during the event and serves as the official record of the incident.

We have a similar process for triage, recovery, and review, mainly utilizing Slack for our chatops. We use Slack, GitHub and Trello as our three main teamwork tools.

But I have _not_ found chat history to a convenient record of what happened. How do you go back to _just_ show a specific event in history? Also, when there is serious downtime, the channel gets flooded with engineer chatter, and it's difficult to quickly comprehend the root issue, decipher what has been attempted, and know how to help.

In addition to our synchronous chatter in Slack during an event, we open a GitHub issue where we post periodic updates about specific things that have been attempted to fix the problem, graphs from server activity during the event, etc. We'll use checklists to keep track of things that still need to be investigated and what has been done. Sometimes, yes, you need to post in both the GitHub issue and the Slack channel, but it provides a much more digestible, high-level view of an outage.

Additionally, the GitHub issue serves as an ongoing record where you can post the content of the blameless post-mortem, reference follow-on work that was done to harden the infrastructure, etc.

Finally, since the GitHub issue is labelled as "Downtime", it's _actually_ convenient to scan through open and past issues and learn from past mistakes and see what can still be improved.

I wish there was real world analytics

What do you mean?

Like those things that beep when you walk into the store but more silent ones making a heat map of your whole store with other analytics

Reply on news.ycombinator.com