A Richter scale for outages | Hacker News Reader

A Richter scale for outages

interconnected.org

37 pointssmcgivern16 comments

Corporate America often has major issues that even internally they can't decide where it rates on the scale - and corporate communications is not about to let that information out. Outright outages for major corps or infrastructure is rare, but performance degradation is so constant for some companies that stumbling along in constant fire-fighting mode is considered normal, IE: something like a 4.0 on this scale is every day. A significant portion of my time over the last ten years has been consulting to large enterprises to solve their production problems and try to ensure they don't happen again. It's shocking how broken things can be before a dime will be spent to improve anything.

I get super concerned when I hear reports like this -- what I'd like to hear is that the Netflix chaos monkey method is being adopted for critical infrastructure to increase resilience. Instead I can totally imagine a software fault that brings transit across a city down for a few days. I think we had a software fault in UK air traffic control recently that knocked out flying for about half a day -- who knows how much of this is going on

Why would "collapse of minor network requiring rebuild. e.g. recent Sony hack that meant no computers, printers, or existing network infrastructure could be re-used without manual check of each item." rate a 6.0 (affect a single organization) whereas "Minor network freeze but can be recovered with a reboot; broad human inconvenience without threat. e.g. regional ATM network down for a day, cellular network down for a day for single operator." is 4.0 (affect many members of the public in a region)

Good point. My examples should ideally be the same scale. For 4.0, I think it's about a single point of failure that, when rectified, recovery is simple. For 6.0, some kind of permanent and cascading problem. The difference between the server being down, and accidentally shipping an update that bricks all clients.

> Like the Richter magnitude scale

Obsolete? The Richter scale is historically important, but in modern usage has been replaced by moment magnitude [0].

As others have pointed out, in any case this subjective scale has more in common with the Mercalli intensity scale [1].

[0] http://en.wikipedia.org/wiki/Moment_magnitude_scale

[1] http://en.wikipedia.org/wiki/Mercalli_intensity_scale

> 2.0 ... Facebook down ...

For some users, this more like a 9.0. We've literally seen people call 911 because they can't access Facebook during an outage.

Author of linked blog post here -- would love to hear what you think!

It's a very interesting concept, because now in 2015 I've seen more and more things using technology and computers where they shouldn't be used. People seem to not understand that the gimmick feature that needs a computer just adds a billion moving parts (one for each transistor, statement of code, etc) that may fail.

I was in the process of buying a new TV and I couldn't find a good ol' TV, nope, all there was "super awesome smart voice commanded can see the weather TVs"... so I bought one intending to just use it for cable (I still used netflix in my computer) and after a whole damn day of updating firmwares and waiting for it to boot (I miss the CRT days)... some capacitors that were in the backlight circuit blew. Thank you very much Samsung. (Btw another clear example of misusing technology was yesterday's post about trains using GPS for "smart door controls")

Oh also, the scale in your post is more of a Mercali than Richter.

I didn't realise there was an effect-focused earthquake scale, that's definitely more what I meant

Logarithmic scales make sense for things with clear, quantifiable measures in some particular dimension. This scale both includes a large number of qualitative measures that aren't particularly obviously different orders of magnitude and conflates severity of effect and time to fix.

I'd love for systems theory (cybernetics or whatever) to have gotten to the point where we could have an actual measure of disruption. Severity seems to come from both what the outage is, and also from our failure to build resilient systems... it's earthquake tremors vs whether we followed the building codes. But unlike earthquakes, it's really hard to tease those two apart

There's one like what you say: Mercali.

An example that sprung to mind while reading was the arson/suicide attack on Chicago's air traffic control facility late last year

http://en.wikipedia.org/wiki/Chicago_Air_Route_Traffic_Contr...

I like the idea for internal discussion of problems but I don't think we'll see usage of a scale like this on say a corporate support Twitter account.

It will end up with the Hacker groups of the world bragging about the 10s they've caused major technology companies and competing for scores...

Would a possible example of a large magnitude event be the 2012 Virginia Derecho[1] ?

Does anyone remember this evening as a day of large outages? I wasn't able to experience the event as an internet user having lost power. This is the type of event I would EXPECT to get a big magnitude.

[1]http://en.wikipedia.org/wiki/June_2012_North_American_derech...

that's the kind of thing. the one that comes to mind for me is the 9 hour AT&T long distance outage of 1990: http://www.mit.edu/hacker/part1.html -- it's these kind of emergent outages that really interest me

Reply on news.ycombinator.com