Death and Valor on an American Warship Doomed by Its Own Navy
features.propublica.org
features.propublica.org
The broken email system had a “major impact” on the ship’s day-to-day operations. Microsoft Outlook did not work. Nor could commanders communicate over a classified email system...
Technicians were constantly fixing the SPS-73, the other main navigational radar on the Fitzgerald. Sometimes, the radar would show the destroyer heading the wrong way. At other times, it simply locked up and would have to be shut down...
A third radar, used for warfare, was slow to acquire targets, but technicians had installed a temporary fix that became permanent. “Problem known since 2012. Declared hopeless,” read notes attached to the repair report...
Other equipment had been written off, too. The so-called Bright Bridge console was supposed to help the bridge crew by sharing information from the combat room. The console had been scavenged for spare parts, leaving the station unmanned.
I found this section of the article legitimately infuriating. How the hell does a first-line warship of the United States Navy get into a state like this? I can understand the occasional malfunction, but this sounds less like that and more like some sad-sack Russian ship circa 1995, where the crew is struggling to keep the thing afloat with just duct tape and bubble gum. But at least the Russians in the '90s had the excuse that the government that had built the ship had collapsed and the replacement hadn't bothered spending a dime to maintain it. What's our Navy's excuse?
Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew, which is shameful. Things don't get this bad in the absence of some kind of systemic rot. If that rot isn't tackled now it'll end up having to be tackled after we lose some future war, in which case, God help us all.
A functioning local email system requires very little resources. This is really an indictment of M$, that they couldn't set up a server that would just run for a couple of decades. If sendmail had been installed in the mid-90s when DDG-62 was commissioned, it probably would still be functioning.
Yes we know that it would be better to receive regular software updates, but in their absence the system shouldn't just shit the bed.
No, it really isn't. For two decades all across the world people have kept Outlook running well enough to function in all sorts of scenarios, including those more difficult than a military ship at sea.
It's not like Thunderbird and Alpine are magically immune to having corrupted data stores and invalid configurations. This is entirely due to lack of proper maintenance.
And yes, postfix or Exim or qmail, once set up, in a fixed configuration like boat only email, could easily run for decades automatically.
I have a sneaking suspicion that the failure was something generic like running out of disk space, which nobody had both the authority and time to fix.
Direct from a deployed friend's mouth:
"When laser printers would run out of toner on base in Iraq, we wouldn't order more toner (we couldn't) - we'd order another entire printer and use the toner cartridge in the new one, and shoot at the printer on the range."
Your tax dollars at work.
1. Toner not in order system.
2. Previous orders of toner found to be incompatible with printer.
etc.
"Never attribute to malice that which is adequately explained by stupidity"
For instance, "Oops I accidentally forgot my briefcase full of classified documents on the bus" is as old as spycraft itself.
"I'm sorry sir, I kicked in your front door and started looking for cash and valuables because I was drunk and thought it was my house! I made a huge mistake!"
I'm not see'ing it, and a quick google search isn't finding any support for that idea; it seems to me quite reasonable: Given 2 common possible explanations, prefer one. Perhaps the "Never" is at issue, but I doubt anyone ever actually interprets never as never.
In fact it's often wise to act as if one sees stupidity, even when one suspects malice. In that way one may tempt the malicious into overstepping. But HR doesn't say anything about actions, only about attributions.
The reality is that big military bureaucracy is among the dumbest of organizations and is capable of feats of illogic almost impossible to appreciate.
Would it be ok for me to say that Theranos failed because of stupidity because halons razor? All i know is the company failed and i don't have time to read any articles so aren't i justified in saying it was stupidity not malice?
So many people are involved in the decision making process it’s nearly impossible to make a decision about anything. When I hear speculation about a conspiracy involving printer toner in Iraq, that sounds like nonsense without evidence.
Theranos is the opposite. A naive young founder pretending to be the reincarnation of Steve Jobs with a manipulative guru/mentor/lover pulling the strings is like a Petri dish for corruption.
The reason is easy -- capex for new equipment is easier to get and more nebulous. The new printer may have started life on a budget spreadsheet as a truck.
Opex is hard. It usually works like this: Some poor clerk had no fricking idea how many toner cartridges or replacement staplers were needed, so he wrote down a bullshit number. Some upstream beancounter patted himself on the back and zeroed out the supply budget to make his powerpoint look good, because he screwed up and didn't order enough bullets or whatever.
Now you're stuck, and whomever orders stuff orders a printer to get toner, which is ultimately cheaper than getting some sort of special clearance to spend opex money that wasn't budgeted.
Command negligence at very high levels, a high optempo without adequate attention to readiness and discipline.
> Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew
Unless you are including the chain of command above the ship's commander up through and including the 7th Fleet commander as “the crew”, you are wrong.
The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces.
This is good to hear, thank you! I'm glad that accountability for these kinds of systemic problems doesn't stop at the captain's chair.
The Navy inquiries determined that there had been widespread problems with leaders regarding shortfalls in training, manning and equipment in the 7th Fleet. The Navy fired admirals, captains and commanders, punished sailors and criminally prosecuted officers for neglecting their duties.
Adm. John Richardson, head of the Navy, called the two collisions “avoidable tragedies.” The ships’ commanders and their superiors, he said in a written statement to ProPublica, were responsible for the results.
“The tragedies of USS Fitzgerald and USS John S. McCain reminded us that all commanders, from the unit level to the fleet commander, must constantly assess and manage risks and opportunities in a very complex and dynamic environment,” Richardson said. “But at the end of the day, our commanders make decisions and our sailors execute and there is an outcome — a result of that decision. The commander ‘owns’ that outcome.”
As an aside, the use of illustrations in this article are really great, from the collision graphics, to the busy sea routes.
One of the decisions not practically available to the captain is to stay in port because he declares his ship not ready.
If a crash happens, like it did, he's legally responsible and his career ends.
But if he declares his ship unfit, then he's also declaring himself unfit, because he didn't ensure that his ship was ready. And under the "operational pace" that's been described in all the articles about these incidents, you can easily imagine the unspoken order that "you will not take your ship out of service." Even though that also is his legal responsibility, if appropriate.
We've all seen or experienced "get it done, just don't tell me how you do it."
My assumption is if all ships in the 7th fleet took themselves offline until training and resources were merely acceptable, there'd be no 7th fleet.
And we're not even at effing war! At least not in the Western Pacific. Imagine the shitshow that will be a shooting war in the Western Pacific.
It's on the leadership above the captain to back him up if he declares the ship unfit - or unable to get off-ship support to make it fit. Having a culture where you can't "stop the line", to borrow a Toyota process term, for serious problems - will lead to serious problems accumulating.
At least some heads rolled for it. But based on what I'm reading here, the problem is systemic.
I would attribute it to a few things. Mostly the over-extension of resources and people. The operational tempo in 7th fleet is absolutely ridiculous, so everything that isn't "mission accomplishment" gets pushed to the wayside. Training, making sure equipment is properly fixed.. our days were so jam packed full of bullshit that this important stuff suffered.
I considered myself lucky if I got 5 hours of unbroken sleep a night while at sea, and most of the people I worked with were the same way. Walking around functionally drunk doesn't lend itself to good problem solving skills. Do you know after these incidences there was massive effort to train people about the importance of sleep? Guess what, this mandatory training forced people to interrupt their sleep to attend.
All in all 7th fleet suffers badly from undermanning based on the operational requirements. Ask anybody. AMA I guess.
Subs are air conditioned, so they can stand either 6 and 6s or less commonly 6 and 12s. Their lives are generally worse in every way. Except they all like smelling each other farts and sharing beds, so it works out.
But I doubt that the culture described in TFA, and this thread evolved in any way but organically.
It's their job to make sure the U.S. military is functioning well, and they're apparently not doing it.
I think it's tough but doable to sustain it at the 100% level, but that would require a whole overhaul of the way the Navy does business.
It is pretty silly that they didn't seem to realize that the radar wasn't tuned correctly. I wasn't a surface watchstander so I can't comment on the radar aspect in more detail. Normally inputs are taken from a variety of sensors, not just one busted radar.
Repeat every 3-8 weeks randomly with gaps of 6 months every 2-3 years:
2 hours of leaving port
1-3 weeks later: 2 hours of entering portIn a main shipping channel near a busy port? Very common.
> It's hard for me to understand how they didn't realize they weren't getting useful data from that radar and switch their primary navigation driver to the other radars, lookouts, the commercial position broadcasting system, or the infrared camera, all of which seemingly worked.
Or the old standby, the Mark One eyeball. As I commented elsewhere upthread, the thing that jumped out at me was not having a lookout on both bridge wings (port and starboard). Even with a ship that's shorthanded, that's the last place you should give up personnel, especially when navigating in a crowded shipping channel at night. They should have made up the shortfall from somewhere else.
Personally, I'd say that if there's so much work to do that between work and eating everyone is up for 19 hours, then you're undercrewed.
But wait! According to the 3M Navy planned maintenance system you have a spot-check with the Captain in the middle of your 7 hours off. Too bad. But wait! The XO has recently been displeased with the cleanliness level of the ship, so as a result during sweepers all people not on watch have to be sweeping the passageway. Too bad. But wait! The berthing has been dirty lately so when your people are supposed to be sleeping they are being woken up by the cleaning. Too bad. But wait! Half the team is on watch, and conducting flight operations requires us to realign a vital system on the ship. Too bad, looks like the people who are supposed to be sleeping are going to be woken up. But wait! We have to have a division-wide meeting and training for reason X, Y, and Z. Ideally you would schedule this at watch turnover but if it's something involved we can have our watchstanders being distracted so just do it in your offtime.
You get the idea.
Senior military officers self-select for being the kind of people who say they can do it. The system deselects people who say it can't be done.
Keep going and going until disaster hits to make everything reset; traditionally, a big war does this. At that point, people do start saying when things can't be done and it gets very hard to sweep failure under the rug when it involves being occupied by foreign powers and people's children never coming home.
It sounds like this is uncomfortably close to the truth about the US situation too; the wars in Iraq and Afghanistan had unlimited money to waste, but this must have been at the cost of other parts of the military. And the political system is increasingly dysfunctional above it.
The navy as a whole might want to save money, but the parts that make it up want to spend as much money as possible; if they don't, their budget will get cut since they don't "need" the resources allocated to them.
What people sometimes forget about crude Russian engineering is that bubblegum and toothpicks are enough to get the thing operational.
Simplicity was always the Russian priority because even in the Soviet days they were broke, and even if they had the money to buy parts getting them to a frozen or muddy air(literal)field in the middle of the middle of nowhere was a feat in itself.
The US military-industrial complex has functionally unlimited resources, so the things they build require unlimited resources to operate and function.
And agree with your point.
My question was more rhetorical :)
You know what DARPA should be doing? Instead of trying to make 'smart bullets' maybe they should consider that complexity and security of IT is a goddam problem and support products instead of just technology.
Someone to make a version of Linux, hardended/simplified it, and then did the same for a few other pieces (i.e. networking gear) and then made 'mil grade' versions of a bunch of things.
And then teach every single member of the military how to do basic stuff, i.e. 'know their computer'. Super basic 'MILINUX' skills, so that weapons systems techs could at least fathom what on earth is wrong with their thing and possibly fix it.
Same with mail.
There is no reason in 2018 that e-mail can't be durable and available for relatively low cost in some system.
Also consider now that there should be a lot more IT dudes in the forces.
Armies are in a boom and bust cycle. When the last war is over they lose their funding and priority over other stuff. Until the next war begins.
This American Life had a good segment in late 2017 about the Navy accidents here: https://www.thisamericanlife.org/634/human-error-in-volatile...
Just a random comment:
She may well have panicked, but depending on proximity, that may have been the correct move, if executed correctly and in time. In other words, the move may have come from training, and it may merely have been inconsequential if executed too late.
A ship turns by moving its back end left to turn right, and vice versa. That's how the rudder affects the ship. "Right full rudder" moves the back end left, pointing the entire ship in a more rightward direction.
I've directly observed this maneuver used in an emergency. I was enlisted, on a Knox class fast frigate. We were refueling underway next to an oiler. The two ships steam forward side by side, and lines are sent across, to eventually get hoses across from the oiler to the receiver (us).
One of our junior officers was in command. The captain was also on the bridge, observing and monitoring. I was there in case what I was responsible for broke, otherwise I wasn't normally on the bridge.
The junior officer allowed our ship to drift to the right, too close to the oiler. At some point the propellers of the large ship started to suck us in even closer.
The captain identified himself and declared himself in command, ordered right full rudder, toward the oiler, and full speed ahead. An alarm was also sounded, "emergency breakaway" or something like that. The maneuver immediately moved most of our ship away from the oiler, and we shot past the oiler (they were also executing their side of the emergency breakaway).
Not saying the two incidents are the same, only that "steering toward the other ship" wasn't necessarily the wrong maneuver. But they were doomed no matter what, as just about everything else relevant that could go wrong was or already had gone wrong.
TFA:
Instead, Coppock ordered a move that disregarded the very basics of her training. She commanded the helmsman to gun the destroyer’s powerful engines to full speed and duck in front of the Crystal by heading left.
I don't think this abortive maneuver is well indicated in TFA's "cut-scenes", but it would never make sense as a way to avoid a collision. Unlike the situation you describe in which two ships were traveling roughly parallel courses, the courses of these ships were roughly perpendicular, with Fitzgerald's starboard side facing Crystal's port side. The way for Fitzgerald to avoid contact would have been to change speed (either faster or slower) or to steer hard to starboard while slowing in order to go behind the other vessel. The presence of other vessels might have caused problems with any of those options, but "matching" the other ship's course was likely to cause a collision. If she had enough power to clear the other ship while turning to port, she had more than enough to clear it while maintaining the current heading. Of course, she didn't have enough power to do either, at 0130.
There is no indication in TFA that Crystal was ever plotted as a separate contact from Wan Hai, even after they finally noticed Crystal. If that's the case they really were flying blind, relying on nonexistent intuition to steer through the gauntlet. It would have been surprising if any of the bridge personnel had had enough experience to simply see the right maneuvers by eye at night, which TFA seems to imply Coppock attempted to do. Perhaps the CO or XO would have had that experience, but since they were both in their cabins instead of on the bridge we'll never know...
I'd say that the captain should receive the same punishment that Coppock gets, and the captain of a month before as well. It's difficult to say from the outside whether that should go up the chain, but perhaps it should, just to be safe.
Based on one single incident, that's a fair comment. But it wasn't one single incident. There was one other collision, plus, as the article notes, a number of near misses that should have been critically analyzed but weren't. That's a pattern for which leaders up the chain are responsible.
One thing that jumped out at me was not having a lookout on both bridge wings. Yes, I know the ship was shorthanded, but that's the last place that should take a hit, particularly when navigating through a crowded shipping channel at night. Take someone from somewhere else.
I'm not sure even that is possible, at least not at night in a crowded shipping channel. A lookout under those circumstances should be focused on one field of view, not having to move around.
> teaching someone else while on watch might be marginally possible - if you only had one side of the ship to watch
As I understand it, the person doing the "teaching" was the Junior Officer of the Deck, not a lookout. The JOOD is supposed to be doing multiple things, not be focused on just watching. Her staying on the port side teaching too long was a contributing factor, yes, but only because of the prior decision to accept no lookout assigned to the starboard side to begin with.
As for the culture that contributed to that decision, I'm not sure it was what you suggest as much as a simple failure to understand the impact of not having lookouts on both sides under those conditions. The fact that neither the captain nor the XO were even on the bridge during the transit is another indicator of that, IMO. The article says the captain was tired because of all the training that had been done that day and wasn't on the bridge because of that, but that just means more failure to understand the impact of tradeoffs: if push comes to shove, having more experienced supervision on the bridge during a challenging transit is more important than getting in some more training that day.
Think of any bureaucracy. The training or lack is documented. Paperwork must be done, and the activities backing that paperwork must be done, because that's what gets measured, since the measurers can't be on board to watch.
Sloppy procedures you might get away with, but lack of paperwork gets noticed immediately 100% of the time. And we all want to get promoted, officers especially in the military's "up or out" system.
I read another article, probably here, about this same incident. One of the officers on duty in the CIC was doing her paperwork, in addition/instead of her function at the time. I'm certain this is the norm, because the bureaucracy will be served, and it always expands, never contracts.
I would go so far to say that paperwork will increase even more, directly because of these specific incidents, "to make sure it never happens again."
The incentives are pathologically perverse.
Yes, and that's a higher-up leadership problem. A military unit's primary purpose is to fight, not to do paperwork. The captain of a ship is supposed to understand that, so he can make the right tradeoffs when he has to, and the higher-up leaders over him are supposed to as well, so they can make sure they incentivize ship captains properly.
This, I have a feeling that the real root causes of this problem will only get worse in response to the incidents. Based on reading this and a few other articles and posts and no experience at all in any Navy (but some with other bureaucracies), I judge those to be mainly ever-increasing demands on resources and nobody in the chain with the incentives to say "No, we just can't do that", and ever-increasing levels of paperwork, training, documentation, and other types of bureaucratic make-work to ensure that the last dozen Major Problems can't possibly happen again. It's also not very professional or conductive to keeping good people in when you clearly throw those at the bottom under the bus when they're involved in a screwup whose true root causes they had no power to fix.
That ship was therefore not fit for duty. Not that that would take a ship out of service, these days. They're probably all more or less unfit for duty, based on this and other articles about the same incidents.
Former enlisted Navy, decades ago, made three Western Pacific deployments in and around the area of the article's concern. Not a subject matter expert, and probably never was.
There were multiple failures that allowed the situation to get this out of hand. Ideally, she would have had forceful backup from the CIC.
Of course it should go up the chain, it should go as high as the people making the tasking to the ship Captains when they know the ships aren't ready to execute. Blaming the crew in this situation is discounting the factors that force the crew to operate like this.
But she was the CO no? Maybe im wrong but isn't it the CO's job to lead? Not just expect the grunts to step and be a leader?
It was a revelatory moment for me. I can understand how the Navy wouldn't be excited to disseminate that kind of demoralizing news internally (though I still assumed they would one way or another). But sailors on his ship have sporadic internet access – I was really surprised that word didn't just get around.
I don't use Facebook, so that's probably part of the reason I hadn't heard about it. But also, news about my industry is beyond the last things I'm looking for on the internet.
I just drive. Your brother probably just does his maintenance and stands his watches, and keeps up with friends and family on Facebook, most of whom would not even notice an article about something as foreign as the military.
> Although the Fitzgerald radars did not show them, more than two dozen ships surrounded the destroyer, all close enough to track. Three of them, large vessels off the starboard bow, posed a grave danger to the warship. They were closing in. Quickly.
> But the ships didn’t appear on the combat room’s key radar, the SPS-67, because neither Combs, nor Woodley, nor anyone else, realized that it had been set to a mode designed to scan the seas at a greater distance. With the SPS-67 button taped over, only specialized technicians could change the tuning from another part of the ship.
Not really an improvement over less flashy presentations IMO
Props to Gmail :).
There's also a lot more of them than Secretaries of State and First Daughters, and the people that Congress and talking heads manipulate, us, couldn't sustain our interest past the second or third incident.