Hacker News Reader

I spent 5+ years on a ship in 7th fleet and am all too familiar with what was described above.

I would attribute it to a few things. Mostly the over-extension of resources and people. The operational tempo in 7th fleet is absolutely ridiculous, so everything that isn't "mission accomplishment" gets pushed to the wayside. Training, making sure equipment is properly fixed.. our days were so jam packed full of bullshit that this important stuff suffered.

I considered myself lucky if I got 5 hours of unbroken sleep a night while at sea, and most of the people I worked with were the same way. Walking around functionally drunk doesn't lend itself to good problem solving skills. Do you know after these incidences there was massive effort to train people about the importance of sleep? Guess what, this mandatory training forced people to interrupt their sleep to attend.

All in all 7th fleet suffers badly from undermanning based on the operational requirements. Ask anybody. AMA I guess.

Would you say this is primarily a resource problem? A culture problem with the 7th Fleet? A culture problem in the USN overall?

"Every enlisted man in the Navy only gets 4-5 hours of a sleep a night" seems to be a common anecdote service-wide, including from people I know personally who served.

Also, "nights" are not a real thing, not in the biological sense. In the surface ship nuclear power plant world where there is no air conditioning, you can't stand watch for more than 5 hours, which means 4 5s and one 4 hour watch. Think you stand the same watch every day? Nope, not enough people. You normally stand 1 on 2 off (5 and dimes) or one on 3 off (5 and 15s). This puts your sleep window at different spots every night.

Subs are air conditioned, so they can stand either 6 and 6s or less commonly 6 and 12s. Their lives are generally worse in every way. Except they all like smelling each other farts and sharing beds, so it works out.

DOD issue, if not USG. Army had (has?) the same issue as well, 2015 whitepaper on sweeping everything under the rug until it breaks (https://ssi.armywarcollege.edu/pdffiles/pub1250.pdf)

I think the high operational requirements expose a culture/procedural/values problem of the shipboard Navy overall, it's just particular visible in 7th fleet because of the unsustainability of the workload.

Can confirm this is a problem in Naval aviation also.

I obviously haven't done an A/B study, but I would figure that having periods of crunch time (long or short) with periods of sane time would both keep military units in practice for when they do need to be up for three days on four hours of sleep AND let them train/maintain/not crash their ships into other ships.

But I doubt that the culture described in TFA, and this thread evolved in any way but organically.

Cultural Problem IMO. My boss likes to say that the military takes their young officers and train the common sense out of them. Personally I blame the service academies for what amounts to 4 years of legalized harassment. First three years are all about getting harassed. The last year is about dishing out the harassment. Then they get a degree to justify the whole thing.

IMHO the responsibility properly rests on the POTUS, who is commander-in-chief of the military, and on the Congressional armed-services committees.

It's their job to make sure the U.S. military is functioning well, and they're apparently not doing it.

Could you hazard a guess as to what percentage of the 7th fleet's operational work would need to be curtailed to get it back into a more reasonable, sustainable position? Thanks.

Operating as it does currently? I think we could probably sustain operations at 70% of what we do now.

I think it's tough but doable to sustain it at the 100% level, but that would require a whole overhaul of the way the Navy does business.

How common are scenarios where there are multiple ships within visual distance? From reading the account of the collision, it seems like everybody relied primarily on one radar to make navigation decisions, and that radar was either broken, or its operators didn't know how to use it. It's hard for me to understand how they didn't realize they weren't getting useful data from that radar and switch their primary navigation driver to the other radars, lookouts, the commercial position broadcasting system, or the infrared camera, all of which seemingly worked. Unless of course actually navigating around other ships is pretty uncommon.

Navigating with other ships in close proximity is pretty common -- mostly when transiting into port, out of port, and through shipping lanes.

It is pretty silly that they didn't seem to realize that the radar wasn't tuned correctly. I wasn't a surface watchstander so I can't comment on the radar aspect in more detail. Normally inputs are taken from a variety of sensors, not just one busted radar.

So is it like a every day kind of thing or like a few hours every week or two weeks?

It depends on the ship, but given the underway scheduling for DDG's in Japan, generically and loosely:

Repeat every 3-8 weeks randomly with gaps of 6 months every 2-3 years:

    2 hours of leaving port

    1-3 weeks later: 2 hours of entering port

It depends on what the ship is doing. When a ship has to transit a high traffic area it generally takes at least several hours, and possibly many hours. This will happen every time the ship goes into port or comes out of port (the USS Fitzgerald was coming out of port), and it will happen whenever the ship has to transit a high traffic area to get wherever it's going (the USS McCain was transiting the Straits of Malacca, one of the most crowded shipping lanes in the world, when it had its collision). Those things are probably happening more often nowadays than they did during the Cold War, because more of the jobs Navy ships are being asked to do are close to shore instead of out in the middle of the ocean.

> How common are scenarios where there are multiple ships within visual distance?

In a main shipping channel near a busy port? Very common.

> It's hard for me to understand how they didn't realize they weren't getting useful data from that radar and switch their primary navigation driver to the other radars, lookouts, the commercial position broadcasting system, or the infrared camera, all of which seemingly worked.

Or the old standby, the Mark One eyeball. As I commented elsewhere upthread, the thing that jumped out at me was not having a lookout on both bridge wings (port and starboard). Even with a ship that's shorthanded, that's the last place you should give up personnel, especially when navigating in a crowded shipping channel at night. They should have made up the shortfall from somewhere else.

Were you doing such essential tasks the other 19 hours of the day that it made sense to only get 5 hours of sleep, or was much of that 19 hours wasted on bullshit?

Personally, I'd say that if there's so much work to do that between work and eating everyone is up for 19 hours, then you're undercrewed.

Excellent question. I'd love to go into detail to paint a clearer picture but I don't think it would be prudent. So let's work with this thought experiment which is definitely not out of the ordinary: We're in a condition where half of the watch team has to be standing watch at a time. As a result, we have a rotation of 5 hours on, 5 hours off, 7 hours on, 7 hours off. In addition to standing watch, training has to be conducted, the ship has to be swept, maintenance of equipment needs to get done. As a result during your 5 hours off you try to get this all taken care of so you can sleep during you 7 hours off.

But wait! According to the 3M Navy planned maintenance system you have a spot-check with the Captain in the middle of your 7 hours off. Too bad. But wait! The XO has recently been displeased with the cleanliness level of the ship, so as a result during sweepers all people not on watch have to be sweeping the passageway. Too bad. But wait! The berthing has been dirty lately so when your people are supposed to be sleeping they are being woken up by the cleaning. Too bad. But wait! Half the team is on watch, and conducting flight operations requires us to realign a vital system on the ship. Too bad, looks like the people who are supposed to be sleeping are going to be woken up. But wait! We have to have a division-wide meeting and training for reason X, Y, and Z. Ideally you would schedule this at watch turnover but if it's something involved we can have our watchstanders being distracted so just do it in your offtime.

You get the idea.

Been there. Knew my time with the Navy would be coming to an end after three straight nights of 2 hrs sleep. Followed by a maintenance evolution. Finally got a chance to get some decent sleep, but was racked out again because I had left a small discarded tube in the work area. I think back and wonder how the Navy was able to function at all with all of the chaos going on during a normal work day. So much of the chaos was self inflicted.

I really don't understand why the operational tempo is so high given that it is peacetime?? Surely some "missions" can be sacrificed as the Navy was clearly not coping. It sounds to me like one possibility is that senior "leaders" wanted to keep increasing the tempo to make themselves look better on paper..?

If it's anything like UK Forces, it's not so much the leaders wanting to look better on paper as the powers that be simply always ratcheting the tempo up, and conversely training, maintenance, downtime, personnel numbers and budgets down.

Senior military officers self-select for being the kind of people who say they can do it. The system deselects people who say it can't be done.

Keep going and going until disaster hits to make everything reset; traditionally, a big war does this. At that point, people do start saying when things can't be done and it gets very hard to sweep failure under the rug when it involves being occupied by foreign powers and people's children never coming home.

> How the hell does a first-line warship of the United States Navy get into a state like this?

Command negligence at very high levels, a high optempo without adequate attention to readiness and discipline.

> Based on the article it sounds like the excuse they've settled on is just blaming everything on the crew

Unless you are including the chain of command above the ship's commander up through and including the 7th Fleet commander as “the crew”, you are wrong.

The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces.

> The cluster of 7th Fleet incidents (the Antietam grounding and the Fitzgerald and McCain collisions) resulted in at least the dismissal of the 7th Fleet commander, the Task Force 70 commander, and the Destroyer Squadron 15 commander, and the early retirement of the head of Naval Surface Forces.

This is good to hear, thank you! I'm glad that accountability for these kinds of systemic problems doesn't stop at the captain's chair.

Yup, from the article:

The Navy inquiries determined that there had been widespread problems with leaders regarding shortfalls in training, manning and equipment in the 7th Fleet. The Navy fired admirals, captains and commanders, punished sailors and criminally prosecuted officers for neglecting their duties.

Adm. John Richardson, head of the Navy, called the two collisions “avoidable tragedies.” The ships’ commanders and their superiors, he said in a written statement to ProPublica, were responsible for the results.

“The tragedies of USS Fitzgerald and USS John S. McCain reminded us that all commanders, from the unit level to the fleet commander, must constantly assess and manage risks and opportunities in a very complex and dynamic environment,” Richardson said. “But at the end of the day, our commanders make decisions and our sailors execute and there is an outcome — a result of that decision. The commander ‘owns’ that outcome.”

As an aside, the use of illustrations in this article are really great, from the collision graphics, to the busy sea routes.

> But at the end of the day, our commanders make decisions

One of the decisions not practically available to the captain is to stay in port because he declares his ship not ready.

If a crash happens, like it did, he's legally responsible and his career ends.

But if he declares his ship unfit, then he's also declaring himself unfit, because he didn't ensure that his ship was ready. And under the "operational pace" that's been described in all the articles about these incidents, you can easily imagine the unspoken order that "you will not take your ship out of service." Even though that also is his legal responsibility, if appropriate.

We've all seen or experienced "get it done, just don't tell me how you do it."

My assumption is if all ships in the 7th fleet took themselves offline until training and resources were merely acceptable, there'd be no 7th fleet.

And we're not even at effing war! At least not in the Western Pacific. Imagine the shitshow that will be a shooting war in the Western Pacific.

> But if he declares his ship unfit, then he's also declaring himself unfit

It's on the leadership above the captain to back him up if he declares the ship unfit - or unable to get off-ship support to make it fit. Having a culture where you can't "stop the line", to borrow a Toyota process term, for serious problems - will lead to serious problems accumulating.

I'm not seeing the graphics on firefox :(

That's odd, I read this entire article on Firefox.

This is all infuriating to read as a civilian. This is what we get for all the billions we pour into the military?

At least some heads rolled for it. But based on what I'm reading here, the problem is systemic.

But what you also get is everything that didn't end in disaster, for which you paid a lot less than you should have, because they skimped on maintenance, training, downtime, upgrades and so on.

I understand where you are going but is this true? Did we really skimp on that or did we instead put into place a culture that is causing busy work and ineffective procedures that could be fixed with the people and resources we currently have?

> Russians in the '90s had the excuse that the government that had built the ship had collapsed and the replacement hadn't bothered spending a dime to maintain it.

It sounds like this is uncomfortably close to the truth about the US situation too; the wars in Iraq and Afghanistan had unlimited money to waste, but this must have been at the cost of other parts of the military. And the political system is increasingly dysfunctional above it.

Budget sequestration was a factor. Ships require increasing maintenance as they age, but the military budget was flat or declining.

Both sides have an incentive to over-engineer. The arms manufacturer wants to add as many systems as possible because more complexity = more money.

The navy as a whole might want to save money, but the parts that make it up want to spend as much money as possible; if they don't, their budget will get cut since they don't "need" the resources allocated to them.

What people sometimes forget about crude Russian engineering is that bubblegum and toothpicks are enough to get the thing operational.

Simplicity was always the Russian priority because even in the Soviet days they were broke, and even if they had the money to buy parts getting them to a frozen or muddy air(literal)field in the middle of the middle of nowhere was a feat in itself.

The US military-industrial complex has functionally unlimited resources, so the things they build require unlimited resources to operate and function.

Nothing new really. You should read up on the state of the Navy in 1941 just before Pearl Harbour. Or just before Korea in 1950.

Armies are in a boom and bust cycle. When the last war is over they lose their funding and priority over other stuff. Until the next war begins.

Except the US military never lost priority after WW2.

Most of those systems probably had upgrades or fixes, but due to a combination of availability, funding, or politics, did not get implemented.

Corruption my friend.

Direct from a deployed friend's mouth:

"When laser printers would run out of toner on base in Iraq, we wouldn't order more toner (we couldn't) - we'd order another entire printer and use the toner cartridge in the new one, and shoot at the printer on the range."

Your tax dollars at work.

On the one hand, yes that's terrible. On the other hand, I can totally see several scenarios where this is a reasonable outcome of an unreasonable system:

1. Toner not in order system.

2. Previous orders of toner found to be incompatible with printer.

etc.

It has nothing to do with corruption. See my other comment.

Hanlon's razor at work...

"Never attribute to malice that which is adequately explained by stupidity"

But never forget that malice does exist, and the clever malicious will often conceal their malice behind a veil of plausible stupidity.

For instance, "Oops I accidentally forgot my briefcase full of classified documents on the bus" is as old as spycraft itself.

Even calling it "clever" or attributing it to spycraft is a stretch. Kids learn to "play dumb" before they learn to form complete sentences.

That's right. Even very dumb criminals will often plead that they were dumb, not malicious, even when the circumstances of their capture make such an explanation ludicrous.

"I'm sorry sir, I kicked in your front door and started looking for cash and valuables because I was drunk and thought it was my house! I made a huge mistake!"

I like to remind people that Hanlons razor is a logical fallacy, and really shouldn't be used nearly as much as it is, especially the way it is (in an isolated context)

>Hanlons razor is a logical fallacy

I'm not see'ing it, and a quick google search isn't finding any support for that idea; it seems to me quite reasonable: Given 2 common possible explanations, prefer one. Perhaps the "Never" is at issue, but I doubt anyone ever actually interprets never as never.

Logical fallacies are not a finite set. Those you find listed on places like wikipedia are merely the most popular.

That’s fine and well, but irrelevant to what I said. I failed to deduce the fallacy myself or find anyone else doing it for me, thus I’m asking.

The fact that it includes the word "never" was the clue for me...

Which I responded to. Reading never as never is more of a misinterpretation than anything else.

Even if we accept that (even though a maxim that starts out "we should hesitate to" rather than "never" will be much less rhetorically potent), HR claims to observe a fact about the world: that malice is far less common than stupidity, or at least that it's profitable to behave as if that were the case. I've never seen that observation supported, in any discussion about HR. My own experience of the world contradicts that observation much of the time. Probably not most of the time, but enough to make it a mistake to stop thinking, as HR advises us to do.

In fact it's often wise to act as if one sees stupidity, even when one suspects malice. In that way one may tempt the malicious into overstepping. But HR doesn't say anything about actions, only about attributions.

I would disagree. Seeing corruption everywhere is your brain trying to assign order to chaos.

The reality is that big military bureaucracy is among the dumbest of organizations and is capable of feats of illogic almost impossible to appreciate.

Maybe if you have never spent any time in the military.

Would it be ok for me to say that Theranos failed because of stupidity because halons razor? All i know is the company failed and i don't have time to read any articles so aren't i justified in saying it was stupidity not malice?

I’ve never been in the military, but I have lots of time in massive bureaucracy.

So many people are involved in the decision making process it’s nearly impossible to make a decision about anything. When I hear speculation about a conspiracy involving printer toner in Iraq, that sounds like nonsense without evidence.

Theranos is the opposite. A naive young founder pretending to be the reincarnation of Steve Jobs with a manipulative guru/mentor/lover pulling the strings is like a Petri dish for corruption.

But humans aren't logical entities.

Hanlon's razor should not be employed with situations where money vanishes.

You say an overextension of resources is one of the factors, which corruption would play a role in.

That's more systematic incompetence and happens all of the time.

The reason is easy -- capex for new equipment is easier to get and more nebulous. The new printer may have started life on a budget spreadsheet as a truck.

Opex is hard. It usually works like this: Some poor clerk had no fricking idea how many toner cartridges or replacement staplers were needed, so he wrote down a bullshit number. Some upstream beancounter patted himself on the back and zeroed out the supply budget to make his powerpoint look good, because he screwed up and didn't order enough bullets or whatever.

Now you're stuck, and whomever orders stuff orders a printer to get toner, which is ultimately cheaper than getting some sort of special clearance to spend opex money that wasn't budgeted.

With all of the above behavior being caused by regulations imposed in the name of cost control.

That's also the case for cheap office printers where a new printer is cheaper than replacing the ink. They make their money on the cartridges and practically give the printers away to try and lock you in.

Supposedly those cartriges in those cheap office printers are not full when you buy them, so this crazyness doesn't work out. Besides, a large organization should be able to procure ink at a reasonable price...

On the other hand, being able to unload on your printer with automatic weapons must be immensely cathartic.

And why are fighting ships using MS Exchange? And not some more hardened linux-based, simple UI/system thing that's easier to admin and keep up?

If I had to speculate, I'd say most likely because there are zero Linux vendors willing to run the gauntlet of a DoD procurement process.

There's at least Red Hat according to: http://www.esi.mil/AgreementList.aspx?type=2

I was in the Canadian Army, and buddies in procurement in Ottawa, so I'm roughly familiar.

And agree with your point.

My question was more rhetorical :)

You know what DARPA should be doing? Instead of trying to make 'smart bullets' maybe they should consider that complexity and security of IT is a goddam problem and support products instead of just technology.

Someone to make a version of Linux, hardended/simplified it, and then did the same for a few other pieces (i.e. networking gear) and then made 'mil grade' versions of a bunch of things.

And then teach every single member of the military how to do basic stuff, i.e. 'know their computer'. Super basic 'MILINUX' skills, so that weapons systems techs could at least fathom what on earth is wrong with their thing and possibly fix it.

Same with mail.

There is no reason in 2018 that e-mail can't be durable and available for relatively low cost in some system.

Also consider now that there should be a lot more IT dudes in the forces.

Microsoft Outlook did not work. Nor could commanders communicate over a classified email system...

A functioning local email system requires very little resources. This is really an indictment of M$, that they couldn't set up a server that would just run for a couple of decades. If sendmail had been installed in the mid-90s when DDG-62 was commissioned, it probably would still be functioning.

Yes we know that it would be better to receive regular software updates, but in their absence the system shouldn't just shit the bed.

There are many things that could have gone wrong. Maybe they ran out of disk space. Maybe some critical certs expired.

Those are issues that would arise within a couple of years, not over 20 years later. If the ship doesn't have anyone qualified to administer computers, then archiving etc. would have to be automated. If the ship used to have such personnel and no longer does, that is certainly a problem at a higher level than the captain.

I was wondering if the previous power issues mentioned had borked the exchange database. Still, thats a day or two to fix, not weeks.

> This is really an indictment of M$

No, it really isn't. For two decades all across the world people have kept Outlook running well enough to function in all sorts of scenarios, including those more difficult than a military ship at sea.

It's not like Thunderbird and Alpine are magically immune to having corrupted data stores and invalid configurations. This is entirely due to lack of proper maintenance.

There are phone switches that were made in 1982 that are still running phone service, including 911 calling, for mid-sized towns.

And yes, postfix or Exim or qmail, once set up, in a fixed configuration like boat only email, could easily run for decades automatically.

Poor choice perhaps, given how many security holes there were in Sendmail.

I have a sneaking suspicion that the failure was something generic like running out of disk space, which nobody had both the authority and time to fix.