Hacker News Reader

Comment by qserasera | Hacker News Reader

>When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.

View on HN

A lot of problems with automation vs manual controls is that the line between those tends to be blurry.

It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it.

It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations.

Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact.

In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that.

Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more.

If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.

Sounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.

It's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us.

Electricity, radiation, remote or sealed mechanical systems, deeply layered software.

In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.

Operators may cause slight SOP deviations in emergency situations. Anticipating this in design between human controls and automated function is going to be very hard. I hope we can get voice input or other advanced input devices setup for communication between systems.

The input isn't the problem IMHO, it's the output.

If a human is in the loop at all then it's because they may be expected to make a decision. And that decision is based on information. And that information needs to be presented unambiguously.

One of the smartest things I read on feedback here was to start with a balance-in, balance-out counting system.

It's unambiguous: if the number of things being processed and output is different than the number of things input, that tells me something.

And that's where automated systems get dangerous. It's incredibly risky if instead of informing and shrinking their response space, they expand it in an attempt to auto-correct the issue.

I wonder what would happen in an actual battle with the fog of war when people are pushed beyond exhaustion. What happens when senior command is dead or missing?

There exists a very distinct line between routine/exercise and war.

You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.

Though one could say that this was due to too little automation rather than too much -- the default behavior could be for the ship to steer itself away from another vessel or shallow water unless overridden (setting off alarms that this action is being taken)

My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.

Unfortunately military ships are expensive and have strict schedules. You may be able to drive your car to a shop any day to diagnose a sensor error but military vessels dont have that luxury.

To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree

There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.

The problem with automation is that it works perfectly until it runs into that edge case which nobody thought about. The problem with the real world is that there are infinity edge cases.

I'd like to use this in my .signature.

> there are infinity edge cases.

this is simply not true. there are finite particles in discrete space so there's no infinity here.

I would call it O(N!) with N increasing over time opposed to infinite and essentially something everything technically faces automated or not - it is just those lacking agency cannot change deliberately.

Not because it isn't effectively beyond linear counting or coverage but because it describes how it grows - in different permutations. Especially with actual other actors involved and literal arms races. Sure your expensive plate armor may stop blades, arrows, and even current musket fire but would it stop cannonfire?

All of which 95% of those edge cases are most likely programmed leading to saving humans countless times in the most situations to the point they don't even realize it.

Why do you think flying is so safe? It's because of automation and redundant systems.

Do things go wrong? Yes. But imagine how many things would go wrong if you gave people a 15,000 line check list.

I didn't mean for the automation to be primary - I meant for it to be a llast-ditch collision avoidance system, like the steering wheel nudge on my car.

Humans should still pilot the ship, but if they are going to let a collision happen, then the ship should try to avoid it even without human intervention (unless they press the "I know what I'm doing and want to do it anyway" button after the ship sounds the alarm saying it's taking evasive action)

So the system doesn't have to handle all of the edge cases, just the "We're on a collision course with that ship ahead" case.

What event was this?

Sounds like the US Navy destroyer that collided [0] with a cargo ship.

[0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...

Based on google results, it appears to be an incident where the USS McCain collided with the Alnic MC near Singapore. She was getting off course, and poor UI made it so hard to steer her back that she crashed. Injuries and deaths resulted.

https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...

Reply on news.ycombinator.com