Hacker News Reader

Comment by brians | Hacker News Reader

Not root cause analysis. Complex systems fail in complex ways. If you want to read more about how this is done, try Normal Accidents (Perrow), Human Error (Reason)... but you're going to land on Engineering a Safer World (Leveson) eventually. You can get a free copy from the author's web site: http://sunnyday.mit.edu/safer-world/index.html

There are tech shops that use these sorts of techniques for every major incident, and for collections of related near-misses. If you want to improve past 4 9s reliability, or if you're considering taking traffic that could wreck a life if it availability or secrecy fail, that only seems appropriate.

View on HN

The Navy likes to keep things simple and called it the Swiss cheese model.[1] It’s usually never a single factor, but a combination of factors. The holes line up and an accident happens.

I think it’s also important to emphasize that getting to this level of safety is so much more cultural than technical. The ability to be open about failure and for people to feel safe in communicating to investigators is critical. The Navy would run two parallel investigations into an incident, one focused on Safety, where anything said was confidential and couldn’t be used in the administrative investigation (FNAEB) that could result in career ending consequences.[2]

Though I worry a bit with the constant wars and punishing deployments this culture may be heading in the wrong direction. [3] One of my XOs said, “One generation of Admirals will make their names breaking the Navy, and the next will make their names putting it back together.”

I have immense respect for the FAA and NTSB. They are trying to infuse this safety ethos into the fledgling drone industry. No easy task.

1. https://en.wikipedia.org/wiki/Swiss_cheese_model 2. https://www.airwarriors.com/community/threads/my-fnaeb-exper... 3. https://www.theatlantic.com/technology/archive/2018/10/the-n...

>Though I worry a bit with the constant wars and punishing deployments this culture may be heading in the wrong direction.

Maybe 'constant wars' is already the wrong direction.

I just wanted to reiterate that I've found the Swiss cheese model one of the simplest and most effective communication tools for thinking about and communicating to laypeople how various events happen where they want to attribute 'cause' to a single factor, decision, etc. In my experience it can be doubly difficult to get people to think in this way once culture has got them thinking differently (I despise regression analysis now, because it breeds a layperson culture of both an illusion of understanding, and that individual variables really have individual coefficients that can be simply manipulated).

My first introduction to it was watching a documentary on air crash investigation as a child, and I've never forgotten it since.

I don't even work in 'safety' per se(currently I do analytical and data work for banks and financial regulation, so one could argue that is a kind of safety), but it just keeps coming up again and again because in the real world, most big human disasters are multi-causal chains, because evolution (in a social sense) will breed out those disasters that are both big in their detriments and simple in their causes.

The JAGMAN investigation is the legal/administrative investigation.

The Safety Investigation Board is separate, and privileged.

FNAEB investigates the crew, and usually follows a Class A mishap ($2m damage or hull loss, or death/permanent disabling injury) but can be triggered by near-miss or pilot flathatting etc.

A friend is a retired Naval aviator, said the #1 reason aviators get permanently grounded from the fleet, is refusal to accept responsibility for mistakes that they were clearly responsible for. Things like forgetting to set the altimeter correctly before takeoff, causing a serious near-miss or pilot deviation. Honestly and fully admitting their mistakes, is more likely to result in a FNAEB returning them to the fleet. Asoh Defense comes to mind.[0]

[0] https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_%2...

Thanks for the reminder, three investigations. It’s been awhile and FNAEB was always a terrifying word that imprinted on my brain. Accepting responsibility was definitely a key part of Naval aviation.

That’s a great example of a safety practice that nets out unsafe: better to ask why the altimeter requires setting and the plane can be started without it!

I do wonder about balance in drone safety, though...

...for instance, Zipline is doing amazing things in Rwanda improving the medical (okay, blood) logistics system drastically. They're legitimately saving lives regularly. But what they're doing is practically illegal in the US due to the way safety regulations are put together (although they have indeed achieved a very high degree of safety and work closely with ATC, etc... it's not the wild west). Some of that is logical... Rwanda is a developing country with a great need that overcomes a lot of safety concerns. But doubtless lives could be saved if similar drones were allowed in the US.

Second: FAA regulations have already slowed the development of electric aircraft in the US which has significant climate consequences and thus can indirectly lead to lives lost...

...that all said, I think the NTSB and FAA do a good job. Particularly the NTSB.

The US already has a good blood distribution system so I doubt many lives would be saved by adding drones. Our major problem is convincing enough eligible people to donate.

The US, like the rest of the world, has about a 7 percent spoilage rate. Rwanda, due to the just-in-time drone delivery network covering the ~entire country, has virtually zero. And rural hospitals in the US have more of a logistics challenge than you might think. Quality and access to care suffers a lot, and maternal mortality has actually gone up in recent years. Something like Zipline could significantly help.

Zipline is doing good work. I think we will get there, it’s just going to take time. Acting FAA administrator Dan Elwell gave a great speech at InterDrone last year about the safety aspect.[1] Personally, I think in some ways parts of industry are slowing down progress. State and local will be key players, the FAA knows this, but industry seems to believe they shouldn’t be involved and below 400’ should be like class e & g airspace with no rights for property owners. If the Uniform Law Commission settles on 200’ that could be a good thing. [2] Then assuming the FAA authorizes a system as safe, private property owners could conduct BVLOS flights over their property (i.e. ranches, mines). Or assuming state and local are involved, they could authorize flights over public routes. The airspace below 400’ will more likely resemble class A airspace than anything else and will have to involve state and local for planning. No magical UTM solution will solve it alone. [3] It will be a combination of technology and operations.

[1] https://www.interdrone.com/news/dan-elwell-speaks-to-audienc...

[2] https://unmanned-aerial.com/drone-industry-responds-to-draft...

[3] https://www.utm.arc.nasa.gov/upp-industry-workshop/UTM%20PP%...

If anyone interested in the book, its webpage is moved.

New MITPress web page is: https://mitpress.mit.edu/books/engineering-safer-world

Since the book is open access, the direct link to the book (again from MITPress webpage) is https://www.dropbox.com/s/dwl3782mc6fcjih/8179.pdf?dl=1

Engineering a Safer World was a truly career changing book for me, and I am so thankful for bts and mstone for introducing it to me.

Would you say it is highly relevant for those interested in AI safety?

What's your threat model when you say "AI safety"? Which scenarios are you attempting to prevent?

Reply on news.ycombinator.com