Hacker News Reader

Comment by pjc50 | Hacker News Reader

I don't see why this is attracting so much skepticism when "crash-only programming" and the "chaos monkey" are popular ideas on HN. There really are only two ways to do software reliability:

- "failure is not an option": you don't get to reboot your rocket controller when it has a floating point error, or your Therac when you're doing an X-ray. This involves investing a lot of time, effort, review, and formal validation into getting it right. It's completely incompatible with Agile and RERO.

- "s--- happens": the cost of failure is small and you can just accept it, issue refunds/apologies and move on. Or show the fail whale. This is much easier and cheaper. This environment moves towards powercycling and redeploying software/EC2 instances/Docker containers whenever something happens. You monitor observed reliability and make a commercial decision as to whether it's unacceptable and you need to fix some bugs.

Almost all of HN works in the second area.

View on HN

Crash-only is rather different from power cycling. The unit of granularity in a crash-only metaphor is at a higher level (process, thread, green thread or other schedulable entity), the whole point being that you delegate to a supervisor tree for restarting crashed subtasks/processes to a known good state while maintaining the uptime and integrity of the entire system (i.e. crashes are not typically user-visible). It works really well because in any complex system running on top of so many layers, the state space expands combinatorially so that performing intricate error diagnosis and recovery procedures will often be a losing proposition from the number of code paths you'd need to properly exercise.

Power cycling is different. A system that expects you to power cycle often because it consistently fragments or cannot dynamically update/reread its configuration is broken and a major annoyance. In many ways, systems that require lots of power cycling are as such because they're designed in a way antithetical to crash-only. It's a failure to enforce boundaries and separation of concerns.

Power cycling needs to happen in individual components. In other words sub-systems might power cycle but the user impact should be as small as possible.

OS processes/Erlang processes/Separate Machines(Containers) can reliable do that. Because they isolate fault propagation.

In a shared memory system if something crashes you don't know what the state of the rest of the system is. Maybe on bad client wrote over the memory of other 999999 clients. So just restarting that one thread is not safe.

Another overlooked aspect is to do crash-only right with recovery you need stable storage. That is where you can persist a known good state so you can restore from and continue. That might be hard or easy depending on the environment.

> You monitor observed reliability and make a commercial decision as to whether it's unacceptable and you need to fix some bugs.

Yap. But "you" here is the developer not the end user. When you go to buy socks on a website, it is not your job to monitor and restart their back-end system or switch IPs. They should be doing you are just buying socks.

> Almost all of HN works in the second area.

Because the first is very hard. NASA does it, medical device manufacturers do it. Critical security modules have it. It is very expensive though.

"Crash-only" means that a subsystem should bug out early when it detects a problem rather than soldiering on and potentially making things worse. The bugging out itself notifies a monitor process which can make a decision on how to recover. The crash-only paradigm was popularized by Erlang, which was created to drive highly reliable pieces of telecom equipment. So it's not entirely without merit.

Hi! I did some research before writing this blog post (okay, I spent 5 minutes reading the wikipedia article on power cycling).

It turns out rocket controllers get power cycled too.

source: https://en.wikipedia.org/wiki/Power_cycling

You probably didn't mean it this way, but I'd like to point out that formal validation certainly isn't incompatible with "agile" approaches, but it won't work the way that most people practice it.

At the systems level you often have a mix of both classes (like your Xray example, typically). And there is often a third path "failure has a well defined mitigation path" in the mix, also.

There's a middle ground there where failure is an option, but with lots of redundancy and modularity so that such a failure is isolated and quickly resolved (at least temporarily). This is the Erlang/OTP approach to software reliability.

Reply on news.ycombinator.com