Why RIM still hasn’t found the cause of its world-wide outage | Hacker News Reader

Why RIM still hasn’t found the cause of its world-wide outage

arstechnica.com

19 pointsshawndumas14 comments

In other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes. Some of these take into account equipment failure, and modelling is often done on equipment life cycles to determine replacement and inspection schedules in oil refineries.

These industries also employ strict management of change processes so that an ad-hoc decision or improvisation doesn't cause an incident.

What puzzles me is why you don't see these kind of practices applied in data center operations.

Are data centers really more complex.. then say.. a nuclear plant?

What puzzles me is why you don't see these kind of practices applied in data center operations.

Simply, people are not willing to pay the price. All of that extra planning requires more man hours. It also requires people, who come at significant cost, skilled at working in such environments. People want to use their Blackberries for hundreds of dollars up front and tens of dollars each month, not thousands of dollars monthly.

It can be done, but the market has determined that it does not want it to be done. It would rather accept some downtime and other problems in order to access the technology at a lower cost.

When I wrote my comment I wasn't thinking of lots of up-front planning. I was thinking along more simple lines like root cause analysis using a human factors or equipment taxonomy (much more affective then 5-whys).. and simple logging of incidents for later analysis.

I think some of these kinds of processes can be adopted with small investments in training and change.

Also, a lot of these kind of failures seem to stem from changes at the networking layer.. which should be more planned and tested given their place in the stack (we're not talking about crazy app behaviour).

You'd better believe that companies like Amazon, RIM, and all the others have very strict change management procedures. Changes to critical systems are generally planned weeks - sometimes months - in advance, and even simple changes often require multiple levels of approval. Even then, sometimes something gets missed, a component acts slightly differently than expected, or somebody makes a mistake.

All of the mistakes the article discussed were network related, and yes, when you're dealing with networks like Amazon's or RIM's, where there are hundreds of thousands or even millions of devices connected, they are definitely more complicated than a nuclear plant.

Arguably so, in that app-level data has a richer set of failure modes than even highly-complex physical materials and flows, but I don't think that detracts at all from your main point.

The linked article used the word "unknowable" to describe the failure modes in a complex datacenter. They're definitely "unknown," but it's way too early to just give up on figuring many more of them out.

If a CTO says "it'll cost $50B to reach less than 5% unknown failure modes," that's one thing. But if a CTO says "yeah, the faulre modes of our critical infrastructure are unknowable" they better start looking for a new gig.

>In other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes.

I don't see why they would need a domain-specific methodology. Toyota gave us the 5 Whys, which applies nicely to just about any kind problem.

http://en.wikipedia.org/wiki/5_Whys

My three year-old daughter uses the same technique, she has no idea who Toyota is. Thankfully the root cause usually ends up being "BECAUSE I SAID SO".

Yes, data centers are more complex than Nuclear power plants. Nuclear power plants generally operate in well-known ways and with few unknown interactions.

eg. the work load on a nuclear power plant system does not spike around Christmas

the work load on a nuclear power plant system does not change when there is a video spreading virally

the nuclear power plant control system does not have to support a diverse eco-system of clients, some of them running hostile code.

the nuclear power plant control system does not need to be upgraded very often

Modern data centres and the infrastructure that runs on them operate in an environment of constant flux, this makes them vastly more complicated than a nuclear power plant system.

Nuclear power plants have a vast number of components and a few very catastrophic failure modes but in essence they're very simple. The components don't change (save for the well known ways in which metal degrades under neutron bombardment), and lots of money has been invested to know exactly how these components act under various stresses. The engineers who operate them operate the same system for years and years becoming experts on how their very well defined system operates and know intimately its exact failure modes, and have a SOP manual and various checklists to quickly pinpoint and eliminate problems.

Nuclear power plants are also developed and operated under a very clear singly focused mandate: Prevent at any cost the Nuclear Power plant from entering a catastrophic failure mode.

While I feel that saying that massive systems fail in unusual ways is accurate, I also am concerned that RIM may not have the same technical prowess as a google or even amazon and yet in a way they are competing at that scale.

Further, I guess pulling their users email into their system makes business sense for them, but that has always struck me as an unusual technical choice.

Further, I guess pulling their users email into their system makes business sense for them, but that has always struck me as an unusual technical choice.

I always thought it was so they could provide mail using a protocol that was optimized for battery life. Constantly polling a mail provider for updates is not exactly a good use of power.

They couldn't really expect everyone who runs a mail server to install the necessary infrastructure to support the devices, so the next best thing is to have computers operated by RIM attached to power mains worry about collecting the email and then notify the wireless device only when there are changes. Apple uses the same basic model for their push services.

Many customers already have BES installed. Why doesn't the phone talk to the BES directly? What value is RIM's centralized data center adding?

The common trend among the outages(atleast Gmail, Amazon EC2) seems to be that the queuing of the transaction messages seems to be overwhelming the network thus further making the attempts at recovery futile.

Maybe a solution is to immediately stop all new requests and have servers that record all the transactions to disk instead of redirecting them to overloaded servers.

How can you make people stop sending messages?

How can "recording to disks" make some change? What do you think is being done otherwise than writing some databases?

You have to build this sort of think into the infrastructure.

For example, TCP and Ethernet retransmission strategies and HTTP servers should have reasonable timeouts that result in error responses rather than just letting incomplete HTTP sessions pile up.

The same principles can and should be applied to any communication protocol.

Reply on news.ycombinator.com