The bullshit of outage language | Hacker News Reader

The bullshit of outage language

37signals.com

30 pointssant0sk131 comments

I call bullshit on this bullshit. When my systems go down, I AM sorry if it inconveniences my users. Why is this not OK? It beats the hell out of some stupid-ass whale picture.

It beats the hell out of some stupid-ass whale picture.

But there is a difference between an outage on a free service and a paid service.

The reality is that is impossible to have 100% uptime, or at least it would be prohibitively expensive.

I think 37signals write about that themselves, and how they get away with 95% or something.

I see a certain dichotomy when it comes to the times when things go wrong: from an provider's point of view, downtime is an inconvenience because they now have to deal with an angry client, but from a client perspective, it looks more like the provider is irresponsible and incompetent, than a mere inconvenience (especially when there's money or job security on the line for the client party).

Then again, I think being cold and going for the "we apologize for any inconvenience" line can have some advantages over being personal - it implies you are more objective and reliable.

If you're dealing with a system outage affecting hundreds or thousands of users, it's probably impossible to say anything else but generics ("inconvenience", "issues", etc.).

Just say what went wrong, how you're dealing with it now and in the future, and what you're doing to make it up to everyone that was (potentially) affected.

If someone writes back with a specific incident caused by the outage, you can address it directly then.

All my emails tend to be formal in style even to people I know well - so I would never consider it appropriate to write 'really, really sorry about that guys' to customers.

> All my emails tend to be formal in style even to people I know well Maybe you should queue this on your reading list: http://www.cluetrain.com

But, would it beat a dead horse?

Hmmm. I remember a while back when 37s had a piece of hardware die, they posted on their blog slating rackspace for not dealing with it quickly enough. A while later they admitted they did not have a hot standby for the hardware that failed (I seem to remember it was a load balancer, so kind of important).

So basically I call bullshit on this. How about when you have a failure you don't blame a supplier who was not in the wrong? I'll try and find the blog post.

We shouldn't have blamed Rackspace. Even if it was their fault, it's our responsibility. Our customers pay us, not Rackspace. We learned from that and haven't blamed anyone but ourselves from that point on.

Isn't it a litte high minded to blog about ways to communicate downtime when the last time you experienced some you blamed your suppliers who were actually blameless?

One hopes people learned from their mistakes.

Yep - here it is: http://www.37signals.com/svn/posts/800-what-happened-this-mo...

"Naturally, we’re going to have a long, serious talk with our service provider (Rackspace). They’re supposed to be the best in the business, but in this instance they failed us,"

Then further down in the comments, DHH admits that their LB blew and they didn't have a hot standby:

"It was no doubt a mistake not to have a second load balancer sitting ready"

http://www.37signals.com/svn/posts/698-downtime-notice

From first comment on the post.

This is sort of ballsy, because they're going to have this throw in their face next time they have an outage.

Hopefully next time we have to apologize for something going wrong we'll be able to do a better job than the standard "We apologize for the inconvenience" reply.

here's how a company such as yours can do better (i'm addressing a web app company in general, not 37s in particular):

* reassure users that the service was "merely" unavailable and no data was lost (i think you already do this)

* explain specifically what happened and why to the best of your knowledge (i think you already do this)

* explain specifically what you're going to do to ensure it doesn't happen again, within reason

* if the downtime is extensive, give paying users a free month or whatever of service (i think you would already do this), within reason

I agree that explaining the reasons/remediation is a good idea.

If you pay X for a service, generally you'll have an idea of the service level and value you expect. I think most people are reasonable about this.

Customers that go off the rails (no pun intended) because of an outage in a free/inexpensive service are probably customers you don't want to have.

You met your standard last time:

http://www.37signals.com/svn/posts/800-what-happened-this-mo...

(Although the Rackspace comment might have been a tactical error).

I was impressed with the post today, is why I commented.

I'd suggest thirty-foot high letters of fire

http://en.wikipedia.org/wiki/Places_in_The_Hitchhiker's_Guid...

They already have.

"Also, you should find someone willing to take personal responsibility."

I strongly disagree here. I always hate this scapegoating behavior, and also it irks me when for some public disasters a random person gets fired and then supposedly everything has been amended. I don't care about the people about your company, I just want you to not screw up. On the other hand, everybody makes mistakes.

I agree - sort of. A company should always present a unified front, and no one person should be asked to sacrifice himself (or herself) because of a problem.

Internally, however, you have to make sure you have processes in place to find out what happened so that if someone was responsible, you can figure out where they went wrong and teach them the procedures that will prevent the problem from happening again (this assumes that the person or people weren't willfully causing the problem, of course).

While I agree with David's sentiment, the more info the better . . . it sometimes may be appropriate to have a stock answer when the event doesn't really generate real emotion. http://status.37signals.com/ "21:20 - 21:40AM CDT DECEMBER 22 (3:20 - 3:40 GMT DECEMBER 23), 2008 Connection problems were experienced to Basecamp and Highrise due to a misbehaving web server. We apologize for the inconvenience."

Hmm, to be honest, I personally don't care about the language used in apologizing for problems caused by downtime - downtimes happen to even the most technically savvy of companies (Google + Gmail anyone?) so really, this is like pissing in the wind.

Also as was mentioned on the first comment, 37signals has used almost those exact same words they're condemning in this blog post. I'm not going to say they're hypocrites... but the evidence is pretty compelling.

http://www.37signals.com/svn/posts/698-downtime-notice "We deeply apologize for any inconveniences this may have caused"

http://37signals.blogs.com/products/2007/11/downtime-explan.... "We apologize for any inconvenience this downtime caused your business."

I'd much rather a company that operates services for 24 hours a day to be able to let people know of the situation as it happens instead of apologizing profusely about it the next day in personalized language.

That's just me though.

I don't know what's wrong with having a canned response. Addressing a problem and ensuring folks that it's being rectified goes a lot longer than leaving your customers out to dry or changing their diapers for them.

But, I guess it's easier said than done if you're ordering a bigger meal than you can eat when your approach to running a business is making it about you, and not your customers (http://www.wired.com/techbiz/media/magazine/16-03/mf_signals...).

He left out "thanks for your patience."

This post reminded me of an JoS post here: http://www.joelonsoftware.com/items/2008/01/22.html

At the time I wanted to see where I could apply the 5 why's, but never got around to it.

If the issue is not the responsibility or fault of the person I am speaking with I do not expect their personal apology. If it is clear they are innocent of wrongdoing it seems unnatural for them to slavishly gush about how sorry they are.

well i can't comment on their site? button just goes gray?

anyway... i think outage messages should have generic language, it is a key to moving on or ignoring the notice...

but you should progressively be able to click for more details, down to the point where if you're so curious why don't you help: here's a link to the active outage discussion with the technicians

Feature Request: The ability to tag two posts as identical, and combine their comments.

Reply on news.ycombinator.com