The Three Levels of Software Safety | Hacker News Reader

The Three Levels of Software Safety

bellmar.medium.com

38 pointsmbellotti12 comments

> Safety is ultimately about preventing a system from reaching dangerous states.

This is a fallacy. Whether a state is dangerous or not often depends on the environment outside the system and usually cannot be totally foreseen by the engineer.

I happily damage my engine if I can drive out of the dessert.

Depending on case I’m happy if files can be undeleted - or not.

But of course I appreciate a warning why the state could be dangerous.

Identifying what a dangerous state is is usually left to the person writing the requirements. Often that's a completely different department from the engineers who implement them.

You are lucky to have someone write requirements who weights in these aspects. I have seen too many specs (this mostly happens in business software) where failures are knowingly or unknowingly ignored... because schedule.

I think failures should be designed for (or at least explicitly defined) even if the requirements do not include them.

I think that's a lot more common in software where safety matters. Often such software is developed according to some regulations like ISO26262.

Yes, I have to constantly remind other front-end developers that we don’t need to be pixel-perfect faithful to Zeplin or the story. It is more of a guide and you can always bring up any concerns you have or if you think something should be done differently and we can talk about it.

Every safety analysis has a system boundary and a definition of "loss". In your example, the topmost system would also include you as a subsystem, and the loss you're trying to prevent is injury to yourself. Getting stranded is a hazard that could result in that loss. A mitigation could be to allow you to damage the vehicle subsystem to protect the person subsystem. A formal safety program would try to avoid process escapes such as overlooking a scenario like that. You might still decide not to design for it, but at least you consciously made that decision.

For an article which aspires to clarify safety I would expect a good definition of the term. Is it "preventing a system from reaching dangerous states"?

In my opinion, you should start with defining what harm to humans you think about. Harm due to privacy leaks? Financial loss due to miscalculations? Locked out by your home security system? Electrocuted by your home security system?

I'm working in driver assistance, safety-critical software in cars. Here is a very high level description how we treat safety:

We start with a high level feature, e.g. automated emergency breaking (AEB). Now someone determines safety goals like "don't break unintentionally" and "don't break too hard". Violating those goals would probably mean the driver loses control of the car and thus harms humans. It needs to be a more specific, of course. Breaking too hard is "decelerate no more than 60km/h in 1.2 seconds" if I remember it correctly. Note that all usual features always use the human driver as fallback mechanism. That results in very interesting questions for autonomous driving.

From those high level requirements it traces down to various parts (hardware, software, electric, mechanics, whatever). There might be a requirement like "this device (ECU), never send signal AebBreakRequest=1 for longer than 1.2seconds".

Then the tedious work comes. You think through all the parts and safety goals and wonder: If something goes wrong here, would it violate the safety goal? If you don't find a good argument, then you need to build some mechanism to prevent it from going wrong. Examples: What if a bit flips in RAM? Use ECC RAM then the probability is low enough. What if there is noise on the CAN vehicle bus? We do check sums. How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence. Those question and answer games can become quite elaborate.

Note that the general pattern here is "if there is a fault, will a safety goal be violated." These days Safety Of The Intended Function (SOTIF) comes with a new pattern "if everything is working as intended, is it safe?" This is mostly triggered from autonomous driving and deep learning approaches. Absent any faults like bit flips, how can we be sure that it won't kill pedestrians dressed up as chickens?

People dont usually say it but the goal of ISO26262 is to build something that kills people at a rate lower than once in 10^10 hours of operation.

>> Then the tedious work comes. You think through all the parts and safety goals and wonder: If something goes wrong here, would it violate the safety goal? If you don't find a good argument, then you need to build some mechanism to prevent it from going wrong.

Your team might find fault tree analysis useful - I recommend finding someone familiar with the methodology. A good introduction is NUREG 0492 which you can probably find as pdf.

"How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence."

10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?

I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.

Interesting description, thanks!

>>"decelerate no more than 60km/h in 1.2 seconds" >>"never send signal AebBreakRequest=1 for longer than 1.2seconds"

I'm surprised to see these specified like this. It seems like it would be much more useful and responsive to real world conditions to specify it is G-Load profiles over milliseconds, especially since the effective braking force is dependent on so many conditions of the vehicle & environment.

So, what do I not know about your system and dev environment?

The level of safety depends on the context the software will be used.

Probably the highest level of safety would pertain to situations like the Apollo 11 mission https://github.com/chrislgarry/Apollo-11

But there are other situations where you don’t need such a high level of software safety. eg when prototyping or exploring different possibilities.

Anyone interested in these ideas may also want to take a look at https://mitpress.mit.edu/books/engineering-safer-world and http://mit.edu/psas — both are great resources for learning more.

Reply on news.ycombinator.com