The Worst U.S. Maritime Disaster in Decades
vanityfair.com
vanityfair.com
Turns out, the B.V.S reports were using raw data that was 10 to 12 hours old - and they didn't explicitly mention that. In the case of a rapidly evolving hurricane, it mattered a lot.
> The B.V.S. map included a time stamp that showed when the processing had been completed, but gave no indication of the age of the raw data on which the forecast was based. Davidson knew that all the forecasts were uncertain, and that they sometimes disagreed. But how aware was he that when he looked at the B.V.S. maps he was looking into the past?
[...]
> Davidson dismissed the plan with a thank-you and did not come to the bridge. Evidence suggests that he was still showing a preference for the animated B.V.S. graphics, which indicated the storm progressing more slowly.
If I see a timestamp on rapidly changing data I sure as hell would like to see if there's a delayed effective stamp (most stock market tickers indicate the "effective date" of their data or say clearly there's a XX min delay).
More of a UX issue. The worst possible kind.
(I learned from a burnt hand: We engaged a non-technical but reliable user to check the daily backup log for errors and report any to us. One day we needed the backup and discovered it hadn't run at all for many weeks. Ouch. I asked the user; she said she indeed checked the logs daily and they were fine. She was right: She was seeing the log from the last backup, unchanged every day. My fault entirely, not hers: We should have anticipated the date problem, and we should have utilized someone technically literate enough to understand what they were reading - in this case, someone who would recognize an 'obvious' problem such as the numbers of files and bytes not changing. And we should have tested our backups more often, but that old lesson almost isn't worth mentioning.)
Super frustrating. And you can't even rely on things staying fixed either, you have to review periodically or it will be back to square #1 within the year.
Agreed, and I drive people crazy with my focus on those things. Thorough design and implementation (including testing) up front cost far less than correcting problems later, and they don't add the enormous cost of downtime and other failures.
But ... I've found that human beings, even serious professionals, have a capacity limit for details, and it's not very high; and if it's for an over-the-horizon risk, attention is very limited. That is my biggest constraint, editing down the details, organizing them, automating them, and making trade-offs to reduce them to a point where others don't throw up their hands. Also, it's hard to get the budget for that up front investment in what looks to others like obsessiveness (it's not; it's carefully considered ROI).
So when you show up for your review (I don't know exactly what you do, but I have an impression), 1,000 details might have been addressed but 50 overlooked. or 1,050 details might have been implemented but there was no capacity for the next 100 - resources ran out, something else came up, etc.
So I can see it both ways.
I recently helped a company big enough that everyone here would recognize the name fix a lot of items around monitoring and logging after finding they were running an important production system in such a manner as to be essentially flying blind. Yeah, those fatal errors in the logs just might be important...
It's also important to realize that the captain was confident with the plan- except the storm in question was abnormal. That's an important fact to consider, as it throws the mariners heuristics off deciding how much more attention to pay.
From the article, "The Coast Guard report also noted that “El Faro crew did not take advantage of B.V.S.’s tropical update feature,” which would have provided hourly updates."
So hourly updates apparently were available even if not subscribed to.
KORD 050251Z 26006KT 10SM SCT110 M02/M09 A3019 RMK AO2 SLP232 T10171089 53009
That's the current weather conditions for Chicago O'Hare.
Looks like gibberish to you and me but airline captains know what it means and they know how old the observation is. Captain has ultimate authority on the ship, but also ultimate responsibility. It was his job to know what that weather report meant. And surprising to me if he didn't, since by other accounts he was experienced, organized, and safety-conscious.
I think the difference is that captains are dealing with currents in the ocean and in the air. For the most part pilots are flying above things that will make the wind change in extremely short distances.
or from gCaptain, which is linked elsewhere in the thread as a source of info on this particular incident:
it will take years of sea time and weeks of classes to even get your Able Seaman (AB) or 100 ton master license. If you want 500/1600 ton mate or masters then even more years of sea time and months of classes. All of these requirements are outlined by the Coast Guard, but they change the rules all the time. You are never really done; every five years, when you come up for your license renewal, you will have to take some refresher courses. Sadly, none of this may be completed while you are at work, but rather while you are on your off time. It is expensive to take these classes and while sometimes your company will pay for the classes, room, and board, don’t count on it.
(edit:format)
I think that's close enough.
There's no easy fixes here.
(Obviously, and hindsight is 20-20, hourly should've been the default).
> He went down to his stateroom after his conversation with Schultz, and when he returned to the bridge he said, “All right, I just sent up the latest weather. Let us clear everything off the chart table with the exception of the charts.” Schultz opened the B.V.S. program. As it happened, according to the N.T.S.B. report, because of a software glitch, the map that appeared was the very same map that had come in with the previous download, six hours earlier. The raw data on which it was based was at least 12 hours old.
It sounds like it wouldn't actually have helped if the captain had the per-hour update, because this undetailed software glitch meant the device wasn't showing the updated maps anyway.
The apparent moral of that part of the story is that the B.V.S. charts are past useless and into actively harmful territory, since they may not just be old but instead actively incorrect.
It's unfortunate when programmers overlook something this big.
Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=447547...
Errata to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455021...
Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455022...
Attachment 1 to Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455023...
Oh man. That transcript was the single worst thing I ever read in my whole life.
I wonder if in the maritime world there is the equivalent of CRM (Cockpit Resource Management), as in aviation.
Too many lives were lost due to 1st officer never taking over command when the error in the commander's thinking had already become apparent or the entire aircraft was being flown, effectively, by a single infallible captain and not a team.
Lessons were learned decades ago and at least in a 1st world outfit I would like to believe that no 1st officer would hesitate to utter the words "I have control" when in such stress.
The recommendations for the crew to "issue a direct command or to use a crew obligation statement — a statement in which the subordinate obligates the senior crewmember as part of the navigational team by using language such as 'we should' or 'our route'" is excellent. People who need to "manage up" at their jobs can start doing this today.
An lt takes command from the Captain during a typhoon due to the caltain’s actions/behavior and is subsequently tried for mutiny.
On the front page right now, there are lots of garbage articles, with 500+ upvotes and comments, and any one of them handily enables new downvoters.
Way more people can downvote now, than ever before, and users relish new power. Having a respectful honor system, and some voting ring detection is great, but the truth is, there are a lot more shitty people on this site. What to do about that?
The root cause was a captain who mistakenly reacted off of 12 hour old data and ignored another source of data that contradicted it. Pretty surprising given the lengths the article goes through to impress upon the reader how "safety conscious" he was.
The article probably emphasises the captain's safety consciousnes to prevent people from "blaming the operator", which is an instinctual and often wrong conclusion to jump to.
The NTSB report says: Performance. The chief mate's performance evaluations were consistently positive. His evaluation from June 2015 stated that he was “passionate” about his work and “an excellent instructor for the inexperienced.”
The report conclusions mention several events and dozens of contributing factors.
If you want to think in terms of blame, which is not helpful to avoid future accidents nor for nurturing a safety culture, the report "blames" the ship owner, captain, crew, National Hurricane Center and the coast guard...
Some actions of the captain were a contributing factor, notably bridge resource management/communication.
This is just one serious problem in the maritime industry that I have first-hand knowledge of.
I loved sailing, but I don't do it any more because it's not the work that's dangerous, it's the other people.
All his articles and books seem to be movie-worthy. I'm not entirely convinced reality is that exciting, though.
He writes in a gripping manner and normally seems to have done pretty thorough background research, as well as personal experience.
On the last day of May in 2009, as night enveloped the airport in Rio de Janeiro, the 216 passengers waiting to board a flight to Paris could not have suspected that they would never see daylight again, or that many would sit strapped to their seats for another two years before being found dead in the darkness, 13,000 feet below the surface of the Atlantic Ocean. But that is what happened.
And I thought that name sounded familiar. William is the son of Wolfgang Langewiesche, author of one of the books that taught me how to fly: Stick and Rudder.
"In the cockpit, the situation was off the scale of test flights. After Dubois arrived, the stall warning temporarily stopped, essentially because the angle of attack was so extreme that the system rejected the data as invalid. This led to a perverse reversal that lasted nearly to the impact: each time Bonin happened to lower the nose, rendering the angle of attack marginally less severe, the stall warning sounded again—a negative reinforcement that may have locked him into his pattern of pitching up, assuming he was hearing the stall warning at all."
They must have been so confused and frightened.
The reality in this case was far, far more exciting than you ever want. If anything, the captain had a preternaturally professional calm.
> ship’s anemometer was in disrepair and had been for weeks
Can't measure wind; the co. didn't subscribe to hourly updates; a "software glitch" gave old reports.
And honestly, experienced Merchant officers have seen so much shit, it's hard to get them rattled.
http://www.stormgeo.com/solutions/shipping/on-board-services...
Charting data accurately is really really hard. On one team we had a guy who said charts are for asking questions, not making decisions, and I’ve found that to be a pretty safe default.
What they needed was hourly updates, which they had from the other service they subscribed to, which was ignored by the captain. Despite the staff being fully aware.
And the captain and staff should always be reading weather data from two sources anyway, for redundancy and accuracy.
These guys had plenty of information but over-relied on one which was 6-12hrs out-of-date at the worst time, which no indication it was old data.
These are solvable problems that have little to do with tiered pricing or a cheapness on part of the company. Both by the above redundancy and the weather company dating their data, and fixing software bugs for such critical software.
A better take away might be, for important decisions, you should deeply understand the data(and importantly, it's limitations) you are using to make the decision.
I mean, there seems to be no reasonable reason why that should be the case, he can see (and be notified) of everything they're seeing and he can substantiate his decision by pointing towards the data, and should do so - because in this case the other officers might have noticed problems with that data.
Very apt also for the Air France 447 crash this author wrote about.
2. I was astonished at this: > "It has been reported that a major merchant ship goes down somewhere in the world every two or three days; most are ships sailing under flags of convenience, with underpaid crews and poor safety records."
It's the key reason I don't find commercial marine nuclear propulsion viable.
Similarly, kt means knots as in speed, but the very same kt also means kilotonnes as in dynamite.
It's ironic that we have 1000x better navigational accuracy than a 19th century tallship captain but we're much more careless with the data than he would have been.
Today, we work with atomic clocks, and our instruments fabricate an angle between the z-axis and a 100th order spherical harmonic series that models the shape of the ocean surface. Every few years, we update the model to keep up with plate tectonics. (Not exactly, but you get the idea.) There are a lot more things to go wrong, and not everyone programs in sanity checks to prevent small corrections from causing large errors.
> That 19th century captain measured the angle between the pole star and the local horizon with a sextant;
Which immediately provided latitude. To get longitude, he did the same with just about any other star and correlated the angle to an accurate clock and carefully-prepared tables.
That "accurate clock" business was why longitude was such a difficult navigational problem for so long until John Harrison finally cracked it. (I know this is getting OT but I find it fascinating.)
https://www.newscientist.com/article/mg23731623-300-ghost-sh...
For American loggers, 90 deaths per 100,000 worker-years [2] making it the most dangerous job.
For American taxi drivers, ~18 deaths (of which 8 are murders) per 100,000 full-time-worker-years [2]. Making taxi driver the most murdered profession - and it was worse around 2000, when the rate was an eye-watering 23.7 murders.
As you say, policing isn't the most dangerous profession; the rate there is 11 deaths of which 3.5 are murders.
[1] https://academic.oup.com/occmed/article/64/4/259/1464740 [2] https://www.washingtonpost.com/news/wonk/wp/2015/01/28/chart...
0.75 chance of dying is what you had on the Titanic!
Your number has no relation to reality at all.
The paper does mention that other estimates have reached both considerably higher and considerably lower totals (they quote one author who estimated 13,000/yr in the 1980s!). It's hard to get good numbers because the worst safety records appear to be among flag-of-convenience nations who also have the worst reporting and insurance practices, so a large portion of total fatalities (probably) come from the places with the highest uncertainties. But even among rich nations they did still find it a comparatively dangerous occupation: it's safer to be a Danish-flagged seafarer than most other flags, but still >10x the fatality rate compared to having a job on land in Denmark.
Is there a "sequel" to Normal Accidents about software? Because I'd buy that in a heartbeat.
Ex-Goggler Yonatan Zunger has had some really good recent essays at, variously, Medium, Twitter, the Boston Globe, and Google+ (for which he was chief architect, and despite its general lack of widespread success, I consider his contributions and lessons-learned experience quite positively).
And he's a fan of Perrow.
Bruce Schneier's work is somewhat more focussed on crypto and security, but covers much this ground, especially later works.
Shoshana Zuboff also addresses social impacts of computers and data systems in her work.
I'm not quite following that sentence. The article makes a lot of the "authorized", saying that meant the captain was being ordered to go directly to San Juan and not use the Old Bahama Channel. However, it seems to say that the diversion into the channel was authorized.
The NTSB report notes that, earlier that year, during Tropical Storm Erika, the Captain felt no need to ask for authorization when changing course--he just made the change and notified HQ. This time, with his job on the line and future career in doubt, he seemed less willing to act independently.
The NTSB concluded that they had no solid evidence that the company was applying direct pressure for him to stick to the route, but they did note a whole host of implicit pressures, as well as a relative deprioritization of safety and crew management in the company culture as a whole, that might have adversely affected his decision-making.
Same here, he goes a different way and costs the company money and all of a sudden its not comparing the cost of the diversion with the cost of the loss of the ship and crew, its only comparing it to a safe but tense voyage.
boss: "Yes, authorized."
NTSB: "What do you mean authorized? It's not your decision. Did the captain ask you if he could go the safe way on the way out?"
>Question I would like to transit the Old Bahama Channel on our return northbound leg
He was clearly asking, not informing.
The problem with the response is that it has clearly been treated as a request ("diversion request heads up through Old Bahama Channel understood and authorized") that could have been denied.
It's obviously not an admission of guilt, but a red flag that's worth investigating.
Right, that's the problem. It implies the captain was not free to make the decision himself (as he should have been).
Saying "authorized" was a big mistake. I work somewhere now where I have to be careful using that word because it implies you're taking on the responsibility, and if something goes wrong through no fault of your own, the blame now lies on you where as it wouldn't have if you just simply acknowledged receipt of the question/statement.
Stupid, but that's the world we live in.
People should trust that instinct. Takes a lot of effort to pull away from the brink, but you should do it. 100% of the time.
Some don't pull back because of cowardice.
The rest keep on going because of the sense of adventure. It makes you feel alive.
It seems clear that the culture in the industry is set up to weed out those who want to err on the side of safety, and reward those who err on the side of maintaining their shipping schedules.
>Davidson answered firmly, “I’m not leaving you. Let’s go.”
Fuck me, I can't even imagine what that must be like. And I'm grateful for that.
Apparently some Norweigans agree with me and are doing exactly this. I suspect this will become the future norm once the bugs are ironed out.
https://www.km.kongsberg.com/ks/web/nokbg0238.nsf/AllWeb/0CB...
The only reason it's rather worrying is because I happen to bank at both of the financial institutions on the list. They are both small, local institutions.
Tried opening the link on a couple other devices but was unable to recreate this. The pop-up appears every time I open the link in Firefox on my phone, though.