Navy’s flawed technology set the USS John McCain up for disaster | Hacker News Reader

Navy’s flawed technology set the USS John McCain up for disaster

features.propublica.org

194 pointsthomasjudge144 comments

In a panic people get tunnel vision. When you design HMI's for a living you have to consider. If my brother, sister, grandmother was using this system during an emergency could we guide them to make the right decisions.

The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.

The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station.

The issues with the navigation system are a whole other bag of worms- who knows what the mess was there.

Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this.

Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.

Integrated bridges are being installed all across USN ships, including Mt. Whitney with MSC right now. They're filled with a hodgepodge of HMIs that are all linked to a control system filled with gyros, ECDIS, RADAR, propulsion, etc. etc. System integration is complex and the design work is done by contracted companies. You don't have the first party vendor doing the integration work. It's a recipe for disaster.

> You don't have the first party vendor doing the integration work.

The only reason this should be true is if the Navy is directly providing this service instead. If nobody is coordinating integration, then that's absolute insanity. Integration is the most important job in a distributed system like this, also probably the hardest.

Knowing the military they probably thought they could avoid needing an integrator by writing out tens of thousands of pages of specs, never stopping to consider that the specs themselves may be incorrect or unworkable or that they would be so complex that no vendor could ever deliver a product that exactly meets the specs.

This is pretty typical of military contracts. They like to spread the work around, so different vendors provide different pieces of the system, and it ends up being a hodgepodge that doesn't integrate well and doesn't work well.

The Navy doesn't like to spread the wolk around. Congressmembers like to spread it around to get a piece of the pork.

Integrated bridge systems have been on Navy ships for decades. This is not new.

This reminds me of the Airbus control sticks, where the systemd averages the two inputs if e.g. one pilot pushes forward while the other pulls back. There is a warning but it can be easily ignored, as happened on https://en.wikipedia.org/wiki/Air_France_Flight_447 .

Also, I suspect there may have been at least one person who raised the idea of disabling/graying out the controls if they're inactive, but it was not listened to or even actively opposed because "it's not in the spec" and the one who brought it up didn't have the power to do anything about it. I've worked in environments like that (thankfully not for long) enough to know that these big and highly bureaucratic projects will tend to have lots of "lack of common sense" problems, either because no one bothers to actually think about what they're trying to do, or the division of labour was so fine that no one could. Lots of process, lots of specification meetings, lots of cargo-culting dogma, but no real thought put into anything. The fact that such developer positions are treated as disposable and easily replaceable probably adds to the problem too.

> This reminds me of the Airbus control sticks, where the systemd averages the two inputs

Now I’m starting to understand the hostility towards systemd.

As someone who has been on both sides of the systemd debate, I heartily guffawed at this. Thanks.

As for the topic at hand, at a 30k ft view, having seen the inside of DoD stuff from a ground troop level, and later from DC itself, I mostly blame the good-old-boy network from being too established and greedy. I figure it's always been this way and technology complicating things has just pulled the curtain back, but essentially they do DC/Virginia back-room dealing to get bids and then under-perform in the delivery but don't really care because DoD's budget is massive and they keep throwing money at them. This matters much less while we engage countries that can't engage back like Iraq and Afghanistan, but as we shift more towards the return of the tri-polar world and DoD has to start planning for more peer level agitation, these problems need to be fixed ASAP or people are going to die.

There is a reason a handful of counties in Virginia had the highest growth of millionares in America post-9/11. I'm tired of contractors wasting money while the real military gets shite, and while maybe some of that is bias from my Marine Corps days (The Corps was notorious for "hand-me" down equipment and small budgets compared to Army, etc), I still say it's about time we financially body checked the good old boys and gave our boots on the ground better treatment.

Hell, how about just not having boots on the ground in the first place in unconstitutional bullshit wars, but I digress.

I have never understood why averaging the stick inputs makes any sense (who is controlling the aircraft in that case?)

In this case, I see that the ship's left and right power controls were transferred from one station to another one at a time. Regardless of whether, in the system's byzantine command structure, there was a way to transfer them together, the fact that they could be transferred individually suggests that someone thought there was a scenario where it made sense to have two different people, possibly in widely separated places, handling the two sides of this function. perhaps someone could enighten me as to the scenarios in which that is useful?

The general use case is Helm controlling the rudders on the left and Lee Helm controlling throttles on the right. They're two separate functions, and while they require coordination to run effectively they don't have to be the same person or the same station or even in the same room.

That's actually the normal mode of operation (split responsibility between Lee Helm on the right and Helm on the left) on a CVN (which uses a very similar system, although with a larger console). It has the ability to control both functions from one side of the console, but that's not the normal mode of operation.

If the bridge ceases to exist, then steering could be controlled from aft steering (directly above the rudders) and throttles from engineering central control (engineering owns the engines). That's a canonical thing, it's the way it's always been -- you have redundancy in those places for many obvious reasons for a warship.

Thanks for your informative reply, but if I am following what you are saying, the Lee Helm is normally controlling all the trottles, and responsibility for port and starboard throttles is not normally split between two people? - though I take your point that, in the case of damage, there might be no single console capable of controlling both the port and starboard power, so I can see why complete flexibility in the assignment of controls to consoles would be a requirement.

That correct - throttle control should not be split between consoles. I don’t believe that’s even possible - it was not in the system I worked on. I suspect the article is not exactly correct in this. The controls normally are ganged so they don’t have to be individually changed but for maneuvering individual screws can be commanded separately on the same console if they are not ganged together.

There's a control location drop-down above each of the two throttles, which implies they can be split.

> I don’t believe that’s even possible - it was not in the system I worked on.

The article says: "As the years passed, Northrop Grumman continued to make improvements to the system. That meant that controls on one ship might not look exactly like controls on another. It was like hopping from the driver’s seat of a Ford truck into a Mercedes-Benz roadster. Sailors could adjust. But it took a while to get a feel for the different controls — and on a fully armed warship, no less."

Split controls, in this case, of the rudder, played a major role in an earlier landmark shipping accident, the 1967 wreck of the Torrey Canyon, the first of the modern age of major supertanker oil spills.

Tim Harford covers this in his new podcast series, "Cautionary Tales", episod "DANGER: Rocks Ahead!":

https://traffic.megaphone.fm/CAD1539409189.mp3?updated=15738....

As described in the episode, there were a number of contributing factors: tight schedule, miscommunications and assumptions between captain and helm, changes of watch, sleep, limited visibility, and the unexpected presence of other traffic.

But at a critical moment, rudder inputs failed to change the ship's course because authority had been transferred to the other set of helm controls.

The thing you're completely missing here is that these systems are built by military contractors, so it really shouldn't be too surprising that they have such awful UI. These companies do not make any commercial products (except Boeing). Their real strength in the marketplace is being really skilled at acquiring valuable government contracts, and being a big enough company (thanks to history and inertia) to actually win these contracts and deliver on them (even if it's late and the product is poor, though it meets all the specs). These companies would not survive if they had to sell their products to consumers. The negative reviews on Amazon would kill them quickly.

>I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry.

You don't need to have experience building ship control systems to know this stuff is stupid. You probably drive a car, right? Can you imagine a car with UIs anything like this? If military contractors were doing car UIs, the roads would be a bloodbath.

Can you imagine a car with UIs anything like this? If military contractors were doing car UIs, the roads would be a bloodbath.

Unfortunately, car UIs have also declined in usability thanks to the touchscreen fad, and very poor software behind them too. At least there's still some choice, but I suspect the majority of new buyers were taken in by a flashly salesman presentation and only later discovered all the problems.

https://news.ycombinator.com/item?id=20190902

Indeed, and some of the worst offenders are among the most trendy car companies (looking at you, Tesla).

It's also worth pointing out that there's a serious lack of talent in the defense contracting industry (not universal -- but certainly on average). The pay is typically not very impressive, the working conditions tend to be stressful (constant cost over-runs, impossible deadlines) and inflexible working arrangements (good luck with WFH). Talented people generally gravitate elsewhere with better pay and perks.

This isn't quite true. The working conditions tend to be less stressful usually, because by law you cannot be required to work overtime without being paid for it. Every hour you work as a defense contractor must be billable to a government contract, and you must get paid for it. You're also not allowed to work overtime unless the customer agrees to it, which they typically only do if there's a schedule crunch.

You're correct about inflexible working arrangements though. WFH isn't very common, and work locations tend to not be in desirable geographic locations.

The pay is generally good, certainly much better than working for the government (though without the job security), though it's probably not going to beat a top salary at a FAANG company by a long shot. But how many other jobs do? There seems to be a trend here on HN to assume that the very top-paid positions at FAANG companies are utterly average for the tech industry, and they're just not.

Overall, defense contracting is fairly lucrative to people who can obtain and keep a security clearance and aren't completely incompetent, and want a job that doesn't have crazy overtime and lets them have a normal family life. But it definitely won't attract the very top-most talented people, because those people can go to work at FAANG companies for a lot more money.

> If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.

But the system was designed so that control of steering or thrust could be taken over from any "inactive" side. If an interface appeared inactive, how would you easily take control from that side?

All the controls would be inactive except for one "take control" button, which when pressed activates all the other controls and disables itself and the controls at the other stations.

looking at these touch controls, I am wondering how one can possibly operate them with any precision in a heavy sea, and avoid clicking the wrong thing. We evolved hands to grasp, push, pull and twist, not to point and tap a glassy surface.

Totally agree. Touch screens are terrible UI for these things.

You'd be amazed at who built this system. I suspect it's easy enough to find out, but I won't throw it out there in case.

I was one of the very early maintenance techs on the system in the 90s. No one with even a foggy understanding of HMI has ever been allowed to touch its design. The issues called out in the article barely scratch the surface.

I'm just a noob front end guy and even I know if the control is disabled...to indicate it.

The idea that this thing behaved that way and made it through a bunch of people is mind boggling to me.

In a sad irony, the article web page itself is a horrible disaster of a user interface.

>Immediate responsibility, the Navy ruled, rested with Sanchez, his officers and senior sailors. They had been lax, even complacent, in their training of the sailors steering the ship. Sanchez had made a critical error in not adding more sailors to stand watch as the McCain navigated the treacherous strait

From what I read consistently about the Navy, and hear from veteran friends, any sailors they could have asked to stand watch would have already been overworked and under rested. Everything I read about the U.S. military, and especially the NAVY, seems to indicate a broken culture of bloat and incompetence from the top down. I think we've been too comfortable as the premier unchallenged world superpower and the services have started to rot from within.

I served in the Navy from 74 to 80, school for the first 18 months and at sea for the last 4.5 years. I worked on a nuclear powered cruiser in the engine room. It was not unusual to stand "port and starboard" watches which meant only 2 people were qualified to stand that particular watch and we alternated duty.

Man, they almost got me for Navy nuke in college a few years ago. The older I get the happier I am that I turned it down.

It provided me the chance to walk into a commercial nuclear power job that paid 6 figures on a HS education.

I think the Army works significantly better. Obviously still wildly wasteful, but they've been under pressure to deliver results for a decade, and actually operate in a war theater, so they actually have processes to get things done, given sufficient money thrown into the hole.

The Air Force is a huge can of failure with the F35. At least they're standing by the goals of having a reasonable number of combat-worthy planes ready to fly, but the refusal to play ground-support roles and use light-combat airframes sets them up to be replaced by an army-run drone program within a few decades.

The Navy, though, hasn't had to do anything more useful than stop Somali pirates for two decades. It shows.

The Air Force didn't design or manufacture the F35.

They didn't go the airplane store and pick one off the shelf either though...

Lockheed gets something like 90% of its revenue from the federal government. They're effectively an arm of the Air Force.

Besides, it's not like the Air Force did no design at all, they came up with a list of wants and needs, had prototypes made by two contractors, refined their requirements after selecting one of the two competing prototypes, etc...

Norway recently lost one of their five frigates[1], and a significant factor was the officers on the bridge lacking experience and being overworked due to training new personnel. Nine of the fifteen[2] safety recommendations made by the accident investigation board were directed to the Norwegian Navy.

[1]: https://en.wikipedia.org/wiki/HNoMS_Helge_Ingstad_(F313)#Col...

[2]: https://www.tu.no/artikler/her-er-konklusjonene-fra-havariko...

I can't find the source [0], but there was another expose'/interview on the USS John McCain where they interviewed darn near the entire crew. My details are fuzzy, but from what I remember the crew has been severely short staffed for years. To the point where 'normal' days were like 18 hours long and sleep for the captain was 2-4 hours a day for years on end. The internal email system for the ship was so broken that they resorted to just using free Gmail addresses, they had to tape over control knobs to keep them from being disturbed, training was completely abandoned, etc. It paints a hell of a bad picture of the Pacific fleet and it's command.

[0] I think it was propublica as well, but I can't find the article there.

Your google-fu failed you because it wasn't about the USS John McCain, it was Propublica's article on the other fatal 7th Fleet destroyer collision in 2017, the USS Fitzgerald. The collisions were two months apart.

The article you are thinking of is here: https://features.propublica.org/navy-accidents/uss-fitzgeral...

Bingo! Thank you!

> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number.

This blows my mind. Can anyone here guess what's going on under the hood? Is this a magic number that a developer came up during testing to avoid running of out memory/swapping?

Just a guess—probably designed using high-reliability / realtime principles. This might involve e.g. allocating memory only at startup, which would explain things. The limit has to be some number when programming this way.

In a real-time safety-critical OS, you absolutely do allocate memory only at startup, but this kind of OS usually isn't used for any kind of GUI interface, it's used for much simpler systems. Another trait of these systems is that the CPU caches are all disabled, because those prevent determinism. But again, this isn't something you'd do on a GUI system; it's something you'd do on an ABS brake controller or an engine ECU, where the amount of memory ever needed is easy to determine in the design.

This is also done in the signal processors that power these systems. I don't know this system in particular, but the USN has been moving towards commodity computing hardware for these systems. Essentially, a rack of commercial servers (usually IBM) running stripped-down RHEL with a real-time kernel. The software for these systems is firm real-time, and generally uses the same principals as other real-time software (including, but not limited to, up-front allocation).

These are just PCs running Windows, and I don't believe there has ever been a thought of real time processing on them.

It's impossible to have hard real-time on an Intel/x86 processor, and "real-time Linux" is not hard real-time.

I doubt it was a magic number, but just an emergent property of how quickly the system could track a certain number of targets. When they saw problems in testing, a workaround was devised, and the 150 number probably came from just counting the number of targets when things started to go haywire. Most likely a bug report was put in the queue, and it never got prioritized because new complaints stopped coming in.

Not to be overlooked: ProPublica actually constructed a 3D version of the navigation system's control board: https://www.propublica.org/article/how-we-reconstructed-the-...

Couldn't find it in your article, but it looks like the 3D models are integrated into the story here: https://features.propublica.org/navy-uss-mccain-crash/navy-i...

Something unclear from the article, but jumped out at me.

It seems that the Achilles heel of the IBNS system was that too much information tended to bork it, but what’s unclear to me (as implied by the article) is if dropping incoming data was actually the sensible thing to do (as is often the case) and if incoming data was prioritized over existing data and that was dropped instead.

The other thing I would point out is that the technical failure is not that a system is complex, but that a complex system has a user interface that allows a sequence of actions to leave it in an invalid or unpredictable state. The transfer of throttle control being incomplete while other actions were taken, for example.

I think the thing lacking here was consistent and sound changes to the state, lack of fuzzing user input to observe this possibility, and the reliance on training to avoid the issues at all. These are problems that can be solved with design.

I haven't worked on any project anywhere approaching significant complexity, but the magic number of 150 supported tracked ships sounds insanely low to me. Is this at all reasonable, or does it indicate the design and implementation were shit?

Does anyone know how much processing is needed here? Is it just updating an internal model of locations and warning about impending collisions? Presuming constant speed and heading that sounds easy, not presuming it sounds not feasible even for just a handful of ships.

Also, if only 150 ships are supported, wouldn't it makes sense for the computer to automatically untrack them? Maybe allow manual prioritization of what to drop first, but surely within a few days of a competent vendor learning adding too many items borked their system they would put out a patch that does the absolute minimum by automatically preventing adding too many items?

> The navigation system could also become overloaded if too much information streamed in from a ship tracking database used worldwide to prevent maritime collisions. The Navy’s second solution was similar to the first: Drop the feed when it became too much.

> They were patches on top of patches that left the Navy’s destroyers without a full picture of the seas around them. But none of the problems was serious enough to attract high-level attention. A Navy system designed to track problems in major ship systems did not contain any reports that mentioned the IBNS until last year, according to a Navy official.

This is what I mean, I have no idea if it’s reasonable or not. Or why the issue exists (bandwidth, memory, etc etc). Or if the “dropped” data is actually dropped at all past 150 ships, and not dropped when it is lowered in priority compared to incoming data.

The number of 150 seems odd as well, but I have no idea what that 150 limit corresponds to in terms of actual data, methods to poll it, update it, connections to leave open, packets to receive, etc. It could be totally reasonable. It could also be insanely small.

My point is, it’s difficult to pass judgement on tech specs out of context.

No inside knowledge here, but in tracking moving objects it would be easy to build in an O(n^2) algorithm when predicting interactions or intersections between ships.

Combine this with the sort of logic used in real time displays (or video games) and as the number of ships increases, the processing time required for the analysis goes up rapidly until you miss a critical time deadline and get your analysis ignored. At which point the system is classified as broken.

Maybe it was due to a weird hardware dependency like the correlation of each ship to a dot on a 10x15 light matrix or something

One would hope that would then have been understood at the planning stage and not discovered unexpectedly once the system was running.

150 tracks without knowing the range of their navigation radar systems doesn't say much. It's hard to put more than 150 objects in a 5-10NM radius. Since it's a navy ship the nav radar is probably more powerful, but still, it's not like a 200NM range air-defence radar. I'm guessing they spec-d the system with incoherent requirements, and either the contractor that won went for the cheapest working hardware without asking too many questions, or maybe they took an old system's spec and asked for an incomplete upgrade... or nobody wanted to go to their boss saying the system didn't work because of some badly written spec or badly managed contractor and we need more money, please ? That would track with leaving the system badly working like this without even having some face-saving 'we're trying to work it out with the contractor right now'.

Or, most likely, normalisation of deviance. The system came at first with so many bugs and problems that when they got it in some stable state, they'd got so used to work around the bugs that they kept on doing that. Look we already put so much effort in this, what's a bit more? Yeah, we're quite happy with the system, at least it doesn't crash every hour like when it was delivered...

It sounded like it was "150 tracks since last reboot" which is absurd, since the workaround was to manually delete out-of-range tracked objects.

Absolutely phenomenal reporting. Take the time and read the entire thing in detail. It’s worth it.

Can any Navy vets comment or speak to these systems? I really worry about the proliferation of these digital interfaces. I’m curious to know how folks feel about them.

I can't speak to that specific system, but I can give you plenty of other anecdotes about the systemic unreadiness of the US Navy.

I was forced to be the on-duty electrician from time to time, despite not being an ship's electrician. One of the responsibilities for the on-duty electrician was to cut the right circuits in case of a compartment fire. Despite my repeated protests that I didn't know what I was doing and needed additional training, I was forced to be the on-duty electrician. Mind you, there were no other duties I had to perform, just that one, and just in case of a fire, so it's not like my every-day workload was increased. Determined not to get people killed in the event of an emergency, I asked the actual electricians what I should be doing. Turns out, they didn't know either. Their plan was, in case of fire, kill power to the whole ship. I don't think I need to tell you, this is exactly the wrong move to make.

There's an ever-growing number of electronic systems onboard ships, and an ever-shrinking number of sailors to run them. Qualifications are a joke in pretty much every department, at least in the surface fleet. Ships are in a constant state of disrepair, it's not uncommon for 30-50% of the communications equipment to be broken at any given time, parts taking months to receive (mostly due to budget restraints).

Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with no sleep, followed by maybe 2 hours of sleep before having to work again the next day, and possibly have another watch from 20:00-12:00. Rinse repeat.

That last paragraph screams of a massive failure in Crew Resource Management. There are acres of case studies about this subject throughout industry, and when people take them to heart lots of accidents are avoided.

Airline pilots are a prime example. How many airliners crash these days? CRM is one of the factors in making them so safe. Truck Drivers now have similar requirements and a much improved safety record as well. This is literally life or death, industries that ignore it are being unnecessarily reckless and unprofessional.

It would be cool if anyone has a source for the improved trucker safety record. It is just common sense, though. I don't know how this no-sleep system ever got traction.

The real issue is that the Navy never wants to blame senior leadership and prefers to blame the easiest targets; they also never want to blame vendors who they might want to buy something from again. The vendors of course never want to take responsibility for problems or they won't ever get another project. Congress never wants to deal with things because it might anger some powerful/wealthy constituent like a defense contractor who holds a lot of jobs. In the end there is no one that will take responsibility to say "this is crap, fix it" because everyone is more concerned with ruining their promotion/contract/influence. So you wind up with ordinary sailors and ship's officers bearing the brunt of this as they are expendable.

I always find myself thinking of articles like this when these collisions come up: https://www.theatlantic.com/magazine/archive/2019/07/future-...

It always seems to me that modern staffing modeling/logic just skips emergency situations especially for critical systems. When I read this idea that generalization and reduced staffing was to replace specialization for exactly this reason. It's hard to moonlight as an electrician during the worst moments, even if you can somehow skimp by at the best moments. The whole point of a navy vessel is to be as self sufficient in all conditions not to have to page a vendor halfway around the world just because the system blew up again, and that's just orthogonal to a business where you can put in a few extra hours to fix a problem or can wait until Monday for a vendor to call back. I'd be interested to know if some people have seen better operations research and modeling to cover these sorts of organizational systems compared to the apparent style of cut a labour force until you see blood and reap the profits from minimizing that expense against long term stability.

The ships referenced in the article are a newer class of ship. It was designed for reduced crew capacity, but as you stated it's simply impractical to try to perform specialized skills outside of your training area.

The real problem is that they're reducing the crews on ships of all kinds, not just the newer ones. I inherited 2-3 people's worth of equipment responsibilities when I checked on board, and that's on top of all the extra nonsense the Navy has you do other than work on your equipment.

It strikes me that at least 16 seconds of screwing around could have been saved by changing the caption of the "Emergency Override - To Manual" button to say "Emergency Override - Take Control." The button's caption would then contain enough information for anyone pressing the button to know that they were responsible for all controls at the station after pressing.

Very sobering article. Makes me reconsider the proliferation of glass cockpits and fly by wire in aviation. Granted, there is no putting the genie back in the bottle, but failsafe and redundancy need to be a central design tenet from the start.

Back in the 80's, I was throttleman aboard a similar sized USN ship. There was no direct link from the bridge to the throttle. Throttle commands were communicated via an indicator and the throttle operator manually turned a big wheel to meet the command. I remember thinking at the time there should be direct throttle control from the bridge. Hindsight is 20:20.

Looks like quite a lot of redundancy here - you can split up controls over various stations really in kind of any way you want. That being the problem. The article led me to think the crash was more a confluence of factors: staff shortages, high operational demands, culture of tolerating cursory training, and a very complex new system that was both substantially harder to understand than traditional fixed stations, and constantly changing.

The complexity of the IBNS system seems to be the ultimate culprit, in that the various members of the stressed and tired bridge crew were each operating under a different mental model of the state of the system. Because of IBNS's flexibility, there were too many possible states for a crew to reason about under pressure.

Software makes it very easy for states to proliferate, but there hasn't been enough research (nor, more soberly, enough spilled blood) yet around designing flexible systems which are easily comprehensible in urgent situations.

Of course this complexity does cause problems in aviation, even with that culture's extreme emphasis on rigorous training, procedure, and human factors. AF 447, for example.

Glass cockpits and FBW are great, in theory. In practice, I don't know because I haven't had to use it. However, don't make the mistake of assuming that just because the US Navy can't get a well-engineered system put together means that Boeing and Airbus are guilty of the same. (Their glass-cockpit systems come from other vendors, who also supply to smaller aircraft makers BTW.)

Also, how many foreign militaries have had problems like this with basic ship steering?

From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Commercial aircraft are not flying into the ground left and right. (The 737MAX is pretty bad, but that's not even related to glass cockpits; MCAS was a Boeing-engineered component, not a glass cockpit sourced from some vendor like Thales.)

> Also, how many foreign militaries have had problems like this with basic ship steering?

> From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships.

Norway lost a destroyer last year after colliding with a tanker. Keep in mind the US Navy is larger than the next 13 largest navies combined. 2 of 3 of these events happening to the US Navy isn't statistically significant enough to say it's a uniquely US Navy problem.

Not to say they don't need to get their shit together.

This story's worrisome in that it makes the Navy sound very vulnerable to electronic warfare.

I mean, if overall operations are so hazard-prone that they're not reliable in the absence of hostilities, how could they possibly be resilient against an advanced actor?

Is the situation as bad as it sounds from the story?

I find it a little ironic that for an article about unintuitive and broken electronic navigation system, the UX on mobile of this article breaks navigation in weird and unintuitive ways.

They value their narrative and slow animated exposition over my ability to scroll and read the article, meaning at multiple times I have to stop and wait for their painfully slow animations before I can continue.

I thought the visuals were well done, but I agree that they should have just scrolled normally on mobile instead of capturing the scroll to advance the animation.

> could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number

> “Usually when we have a fault with that system,” Sanchez said, “their resolution is to reboot the system.”

Of course it is.

Software is going to kill us all.

He ordered Bordeaux to take over steering the warship while another sailor controlled its speed. The idea was to avoid distractions by having each man focus on a single task in the heavy maritime traffic.

I've not controlled anything that floats bigger than a small fishing boat, and I realise the timescales of reaction here are much larger, but that just sounds wrong to me --- changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.

I've not controlled anything that floats bigger than a pedal boat, but to the extent sci-fi shows and navy fictions show can teach me anything, I've always had the impression that the direction and speed of ships are controlled from two different stations on the bridge.

There was a time when the wheelhouse and the bridge were separated, at least in some ships and navies.

See this 1941 British Navy training film:

https://invidio.us/watch?v=ocUD07fvf48

I'm curious why these things are manually controlled at all. Seems like you should give it a heading and it goes that direction, self-correcting along the way.

Because sometimes he "wrong" action is the right thing to do. No automated system can anticipate everything. Think the NASCAR "drive at the wreck in front of you" - by the time you get there it will have moved.

Especially on a ship that might have to spring from "situation normal" to active combat (maneuver/attack/defend all at once) without a pre-knowable trigger condition.

>> changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.

It's interesting you mention this. My uncle and cousins have been sailing for years on Lake Superior. They navigated some of the largest races on the lake for decades, including the Trans Superior in every imaginable weather conditions. They are, IMHO incredibly experienced sailors.

I just sent this very article to them and my uncle responded within the hour about how having two people doing these in tandem was a recipe for disaster because of the very issue you pointed out. He said you should have one capable person who should be able to do both since it can be a delicate balancing act. Too much speed and over correction with steering can have disastrous results.

The coordination between steering and speed is the job of the conning officer, not the helmsman or lee helmsman. Things happen quite a bit slower on a large warship than they do on a sailboat, so that method of control works fine.

The people with the controls in their hand aren't the people deciding what to put the controls at. They're told where to set them, and they do it or report that the mechanisms have failed.

The reason for this is that the people making the decisions have more important shit to worry about. Things are different when your primary concerns are getting shot at and shooting back. In war time it's not atypical to get your engine or rudder shot out, and if you have someone who has one job - to set the rudder - he'll notice and diagnose it a lot faster than someone who's tasked with avoiding incoming fire, bearing weapons on the enemy, etc. Distribution of responsibilities is incredibly important in war. Combat tends to break down careful, considered decision making.

Also keep in mind most of these people aren't seasoned. This dude was just out of high school. Information overload is a thing.

Your relatives don't know what they're talking about. Offshore racing powerboats typically have separate crew members for helm and throttles. Sailboats are completely different.

Button pushing is beneath a gentleman-scholar commanding officer. On the other hand, a single worker-drone with all the controls in his hands might get his own ideas. Multiple stations with partial control, coordinated by the captain's voice, fit the organizational hierarchy best.

So much so that the military even tried to do this with aircraft cockpits, until it was obviously unworkable. The "hotshot young maverick pilot" archetype was allowed to form because they physically couldn't help it.

At least, this is the story told in The Right Stuff.

The advantage of audible instructions is that everyone can hear them. It is easier to catch mistakes and helps maintain situational awareness.

> He ordered Bordeaux to take over steering the warship while another sailor controlled its speed.

That was standard practice back in the day and for many decades before that: One sailor served as the "helmsman" doing the steering and another as the "lee helmsman" transmitting speed orders to the engine room, each as ordered by the conning officer. (Former aircraft-carrier bridge officer here.)

Example: Conning officer: "Right standard rudder, steady on course 090." Helmsman: "Right standard rudder aye, sir [or ma'am]. Sir, my rudder is right standard." Conning officer: "Very well." Helmsman: "Sir, steady on course 090." Conning officer: "Very well." Conning officer: "All engines ahead standard, make turns for 15 knots." Lee helmsman: [Repeats back the order in the same way; rings up "ahead standard" on the engine-order telegraph [0]; and verbally tells the engine room over a sound-powered telephone [1].] Lee helmsman: "Sir, engine room acknowledges all engines ahead standard, make turns for 15 knots." Conning officer: "Very well." The quartermaster on the bridge records the orders in the ship's log, and the watch officer in the engine room does the same for the engine room log.

Don't know how they do things now.

[0] https://www.wikiwand.com/en/Engine_order_telegraph

[1] https://www.wikiwand.com/en/Sound-powered_telephone

Back in the day I didn't get verbal engine order instructions after the telegraph changed. I matched the telegraph pointer and that signaled to the bridge when they saw it move to match their order.

Still done the same way.

CVN CDCWO / TOPWO here.

That's the standard for driving Navy ships and has been for a very long time.

The two driving are under the command of someone senior to them (the conning officer) who gives direction to both and is responsible for ensuring the ship is being driven correctly.

The only thing here that is worse than this shitstorm of a system design is the vindictive attitude of the Navy towards the sailors who have to cope with it. Flogging and hanging from the yardarm may be gone, but the spirit lives on.

Update: I have this suspicion that the Captain was charged with homicide in order to get him to take a plea deal, so that the problems with this system, and the Navy's failure to properly train crews in its use, would not become an issue in a court-martial.

Those look like Motif gui widgets from the 1990's on the touchscreen! https://features.propublica.org/navy-uss-mccain-crash/assets...

These days we are so hyped on the cool shiny objects, but failing at the most basic concept, HSI = Human System Integration:

“ Systems are comprised of hardware, software, and human personnel all of which operate within a surrounding environment. Too often, acquisition systems programs fail to consider the human capacity or requirements as part of the system. This leads to poor task allocation between hardware, software, and human users or supporters. To promote ideal task allocation, it is critical that the human element be considered early in system development.”

Any entry level UX designer would tell you that you can’t pack more and more control into an opaque and confusing system and still lay all the responsibility on the user when something breaks down.

I really wonder, what does UX testing and iteration look like at Northrup-Grumman?

>I really wonder, what does UX testing and iteration look like at Northrup-Grumman?

It's probably very limited, if it even exists at all. What I'm betting happens is that somebody says "The Navy needs a new widget for X purpose!". and the only input the designer gets to have is where it goes.

I'd love to hear how UX designers in research challenged areas such as the military do their testing. It's not like they can just go on UserTesting.com and get feedback from somebody for $10 a test.

>>I really wonder, what does UX testing and iteration look like at Northrup-Grumman?

As someone who worked daily with two different NG-produced command and control systems, it's probably between "non-existent" and "flaming dumpster fire".

https://www.northropgrumman.com/Capabilities/C2PC/Pages/defa...

https://dzone.com/articles/war-fighter-netbeans-platform

Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old version of Google Earth. Newer versions of C2PC are a polished turd, advancing the app's interface from 1995 to 2005.....despite hitting the operating forces around 2017-2018. We had bug submissions on some key mission-critical visualization features in the new version with patch turn-around times looking like 9-12 months at best.

The Navy has a serious, serious leadership problem. Their seals have also been fucking up left and right.

This would all be comical if it wasn't so tragic.

touch screens are a horrible idea for this type of system. I also think they're a bad idea in cars. there is no muscle-memory with a touch screen.

Tried scrolling article up and down. Animated images hijacking the mouse control randomly, showing some motion then freezing. A good illustration.

This is the latest innovation in "interactive story telling".

It is just the ultimate in irony: here is an article crying murder because a touch interface allowed a sailor to un-gang engine power, then slide a big fat green power slider for one engine to half of what the other big green slider for the other engine directly next to it showed.

Meanwhile here we are furiously scrolling to hit invisible "scrolled distance" breakpoints to advance a slideshow.

It's even worse if you're used to using spacebar to page-down.

Meh, I use spacebar to page-down, but am ready to fall-back at a moments notice because so many sites draw their header over the body, so a spacebar will end up by scrolling by more than a page...

The websites that pull this off "well" do so by giving you feedback that your scrolling is still doing _something_. For example, Apple pulls this off reasonably in their marketing pages by moving things in the background: https://www.apple.com/iphone-11-pro/

This one just sticks and does nothing, which is exceptionally awkward and feels broken.

It's even worse on mobile, locking the page up entirely with Javascript. I assumed it was untested and broken there, and shifted to desktop, but it's broken here too.

This reminds me of "Pentagon Wars."

DO NOT LET YOUR FRIENDS/FAMILY JOIN THE NAVY.

-ACTIVE DUTY SAILOR

Openly defaming your employer might have serious consequences.

I wonder was the system coded in Ada?

Reply on news.ycombinator.com