Hacker News Reader

Hacker News

Comment by ducktective | Hacker News Reader

Parent Full thread

ducktective

>344 single points of failure

How can a system have more that one single point of failure?

View on HN

If you're in a car being held above the ground by a chain, every single link is a potential single-point of failure.

What a fantastic comment. In a visual and immediate way to explain such a complex concept in one sentence.

But the chain itself can be considered as one component and it becomes a single point of failure. Or if we go in the opposite direction, then we can breakdown links and say "Every single grain boundary is a single point of failure".

Partially-agreed on the former, but not the latter.

So long as the chain is not loaded near its tensile limit, then the grain boundaries that support the load within each link do so in parallel and are therefore redundant.

The selection of the allegory of the chain was intentional -- each link must be properly formed, or the entire chain will fail. If it breaks, it is surely correct to say, "the chain broke", but in truth, it was actually link-86.

For JWST, the remarkable/audacious thing is that many links in the chain from launch to observation are potential single-point failures. Furthermore, many of them haven't ever been tested independently in space... ever. It is a hell of a triumph that JWST has gotten this far already.

If even a small fraction of the instrumentation works at this point, we are going to learn a ton about the universe, simply due to JWST's position, collecting-area, and mirror-diameter.

You're right. I think I was searching for a better analogy. Abstractly: Sub-system SPOF conditions can be bundled up as a single System SPOF condition by multiplying the probabilities.

Also, I don't want to underplay JWST's success or its challenges. But, when saying 300+ SPOF conditions, one has to specify at what abstraction level. Otherwise, it can be misleading.

Each link in the chain is itself manufactured/created/constructed. If the weld is bad, the link may fail. "every grain boundary" is not formed in the same way each link is. The analogy is solid - actually, it's very good in that it conveys clearly and succinctly the concept the parent was asking about without introducing extraneous concepts. Very much in the flavour of EWD: "The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise."

Another example is a small single engine airplane. The engine can be considered a single point of failure but if you zoom in it’s a complex system with many redundant components (multiple cylinders, multiple spark plugs per cylinder, etc.)

A system may be implemted as a set of cooperating microservices each one being a (single) point if failure of the entire system.

Like many Node dependencies

Contrast a chain with double links, where if any link fails another is entirely capable of holding the load.

In a real chain, only one link is actually the weakest, and it is at that link where the chain will break, in the hypothetical you offer.

Which is true, but also not what's being discussed, which is "single point of failures", not "weakest link".

That’s not correct, someone with a bolt cutter can get any link in the chain and it causes collapse.

Single point of failure doesn’t mean “weakest link”. It means if this one piece (“single point”) fails, the whole system will fail.

But you don’t know which, and others may fail first due to various adverse events (say, random micrometeoroid strike). If any link fails, the whole chain fails.

Contrast a second chain, or double linked chain, so if any link fails the load is not lost.

This assumes the environment is completely predictable.

If you know the strength of every link in the chain with perfect accuracy, and you know that the only potential cause of failure is too much weight being placed on the chain, then the only link that can fail is the weakest[1] one because no other failure can happen before that one.

But really you need to design for the idea that various things might happen. Someone else gave the example of a person choosing a link to cut with bolt cutters. The person's choice is what's not predictable in that example.

---

[1] And if you assume it's not possible to have two links that are exactly as strong as each other.

"only one link is actually the weakest"

Which isn't relevant since (obviously) no one was talking about how many links are the weakest link.

"it is at that link where the chain will break"

Wars have been lost by thinking this way. e.g., D-Day was a successful assault on a strong link.

The classic counterexample to the chain (where each link is a SPOF) is the braided cable or rope, in which each individual strand shares the load collectively, and loss of some fraction of the total strands won't itself cause failure.

That's not to say that cables or ropes cannot fail, but when they do so, failure of many individual components is required.

A recent example is the failure of support cables for the Arecibo radio observatory in Puerto Rico in 2020. This was the result of a progressive failure played out over months, though at an accelerating rate, until the final catastrophic failure and loss of the instrument as a whole. (This wasn't wholly unforseen and was precipitated by a long period of rather intentional neglect of maintenance.)

Even after the failure of two complete cables, the remaining cables supported the load of Arecibo's instrument platform. On 1 December 2020, strands of one of the remaining cables began breaking, at an accelerating rate, leading to the total failure of that cable, then a second and third, and with it loss of the instrument platform. The final moments were captured on video.

Sequence begins at about 55s here: https://yewtu.be/watch?v=b3AASKr_iHc

Thank you for linking to one of the invidious instances. Youtube itself has become so annoying it's no longer workable for me.

I think linguistically the term can be hard to parse.

If any one of 344 pieces were to fail during deployment, then all of the deployment has failed and the entire $10B was a loss. Consider the engine in your car- how many single pieces of it could fail before the entire engine can't work? The difference with Webb is that most of those points were single actions that had to work once.

And we're now at a stage where most of them did not fail.

"A single point of failure (SPOF) is a part of a system that, if it fails, will stop the entire system from working."

>I think linguistically the term can be hard to parse.

Yeah, "solitary" might make more sense ... or "non-redundant". Or "point of single failure".

Single point of failure is a term for non-redundant, critical parts of a system:

https://en.m.wikipedia.org/wiki/Single_point_of_failure

The sequential deployment process had many possible individual points of failure along the way - if any step went wrong then the full deployment failed.

well yes, but wouldn't we refer to them simply by "points of failure" not "single"?

I think it's single in the sense that only a single failure in any of those points would render the entire project a failure, not that there's only one point that can possibly fail.

Contrast, say, a single-engine jet plane with a twin-engine jet plane that can still make it to the airport safely with the remaining engine, should one engine fail mid-flight.

"single point of failure" is a phrase that refers to any subsystem which the whole system is dependent on.

Most failure points on Webb have redundancies. So their failure doesn't brick the telescope, both the part and its redundancy have to fail before Webb is bricked. The single points brick Webb with only a single failure.

Because a point of failure could have multiple redundancies, and in this case they did not, every one of those points was implemented in a non-redundant way.

It's a point where a single failure causes system failure, but that's a lot more words. These things tend to get whittled down to "terms of art", the meaning of which is understood by those trained in the art. It's perhaps best to just call it an SPOF and then people will hopefully google it to find out what that means.

BY the phrase not meaning what you think it means.

https://en.wikipedia.org/wiki/Single_point_of_failure

"A single point of failure (SPOF) is a part of a system that, if it fails, will stop the entire system from working."

I think SPF means a construction where there's no redundancy provided? Though it does raise the question of how this is counted: table with 4 legs, any of them breaks, table is still standing. Table with three legs, any of them breaks, table is broken.

In your body, the brain, heart and lungs are all single points of failure.

If either one fails, you die.

You have two lungs, though. One can fail and you will still live. As with kidneys. Strange that there is only one heart.

It's strange but we're still pretty bad at using more than a single computer core on average.

Lungs and kidneys can work independently, but two pumps running at the same time on the same circuit will cause more failures ... it's like parallel vs. sequential circuits.

Airliners are designed so that no single failure can bring it down.

That's not entirely true. Any reasonably complex system will almost certainly have single points of failure, including airliners. Those single points of failures might be very unlikely, but they are still there. E.g. what if the front fell off?

It is entirely true. Source: I worked on 757 flight controls design for 3 years.

> what if the front fell off?

All structural components are redundant and have redundant attachments. The wing spars are doubled, for example.

There are certainly single points of failure in airliner design. An example that springs to mind is the stabilizer trim jackscrew. There are supporting systems which try to mitigate the risk of failure or the likelihood of failure, but bottom line, if the jackscrew fails, there is no redundancy. You've lost stab trim, potentially making the situation unrecoverable. Of course Alaska 261 is one such example of a jackscrew failure.

I worked for 3 years on the design the 757 stab trim jackscrew. We considered every failure, and had redundancy for it.

The AL flight cracked its nut, and the balls fell out. I don't know the details of that design, but on the 757 the ice scrapers on both sides of the nut will hold the nut in place if the balls fall out.

If I recall correctly, if the AL pilot had simply left the trim system alone after it failed, and landed the airplane, it would have been fine. Instead, he kept fiddling with it, driving it up and down, until it tore the nut to pieces. The loud screech it made when running in a half broken configuration should have chilled anyone's blood to bone at 30,000 feet.

As that incident, and the later MAX crashes clearly demonstrated, once something goes wrong with the stab trim system, you move it only enough to get the airplane into a flyable state. Then, you turn it off and leave it the frack alone. Land it, and let the mechanics figure it out with it safely in the hangar.

One incident where the stab trim failed and the poor pilots were just passengers to their doom was that 747 that took off from the base, and some tank in the cargo hold broke loose, slid back, smashed through the bulkhead and broke off the whole stab trim system.

That really wasn't the stab trim system's fault, you can't really design for a tank falling on it. It was the fault of no redundancy in the straps holding the tank in place.

The video of the crash, shown endlessly on the news, makes my gut turn to water.

I fully agree that the risk tradeoffs made on these designs are highly unlikely to fail. The point is, though, that if they DO fail there IS NO redundancy. There aren't two jackscrew mechanisms. It's a single point of failure. Highly reliable single points of failure don't become redundant because of their high reliability -- they are still single points of failure. And sometimes those must be accepted -- even in safety critical systems.

Tell that to the passengers on the two 737 MAX planes that crashed.

Are these 'points of failure' ? not single points of failure? I assume some of these have redundancy but could fail.

There are 344 individual items that can make the $10 billion satellite be a paper weight in orbit. Those are the very definition of single points of failure.

Reply on news.ycombinator.com