FAA faulted for outsourcing 787 safety checks to Boeing | Hacker News Reader

FAA faulted for outsourcing 787 safety checks to Boeing

seattletimes.com

47 pointsCreate46 comments

The sad/pathetic part of this is that Boeing are infinitely more qualified to make these assessments.

The scary/pathetic part is that when faced with such a problem, the FAA's choices were: 1. Punt the responsibility to Boeing and hope it works out. Or 2. Just tell them "no" as we've been telling everyone else who wants to use whatever post-60's technology they've chosen for their aircraft.

I don't necessarily disagree about those things being sad/scary/pathetic. But these things cost money. Taxpayers are generally averse to spending money on long term prevention. Something that people consider an "expensive boondoggle" and refuse to pay for before an incident tends to become "shame on you for not doing your job" right after.

Do you remember after the BP spill how people were shocked -- shocked! -- that the government did not have deep sea oil rig submersibles and that we had to rely on BP's resources? This type of thinking blew my mind. Who on earth would have supported paying for deep sea utility submersibles for some "black swan" oil spill before this incident happened? And the funny part is, now that it's over and everybody has moved on, they probably still wouldn't.

So back to the FAA... how are we going to pay for top tier engineering talent in the kinds of numbers required to check these components at the level of detail we might wish?

TL;DR: proper oversight costs money.

This is a growing problem in a lot of industries, sort of similar to what Joel Spolsky calls the "Lord Palmerston" principle http://www.joelonsoftware.com/articles/LordPalmerston.html

Everything's so big, but you can't learn any faster than people before you, when everything was smaller, so you focus on a tiny area. In the 60's an amateur mechanic could understand and tweak a car, now they're so complex a profession mechanical engineer would have a hard time.

How many people could have actually understand the issue with the 787 or with mortgage CDO's before anything went wrong? Only a handful, and they're generally the ones making the things.

Checking if something has a 1 in 10m probability of failure, is the wrong idea. The question should be 'What if it does?' and that wasn't adequately answered here. There didn't seem to be any thermal isolation between cells, and a questionably risky type of Lithium-Ion chemistry.

I disagree with the premise that the FAA might understand modern aircraft design better than Boeing. Or that Boeing doesn't have plenty of incentive to make the aircraft very safe.

But you can't disagree that this is a classic example of conflicting interests, and exactly the sort of thing that should be avoided when so many lives are at stake.

Whose interest do you think was served by inadequate battery testing?

I'm pretty sure Boeing isn't laughing all the way to the bank right now.

Boeing's calculation of a battery reliability threshold would be related to cost of manufacturing vs cost of potential lawsuits related to failures. If they are highly unlikely to fail, and the cost of manufacturing them to an even higher spec is more than the accepted risk, they will not do it. Since the FAA cares less about Boeing's profits and more about the public's safety (hopefully), they would likely have a higher threshold.

Safety (ability to cause harm) and reliability (potential to not meet requirements) thresholds are different animals.

I work in a regulated industry (medical) and I can assure you that I have never seen a Safety threshold that was in any way related to manufacturing or any other cost. The thresholds are always set according to how much harm can be caused and the likelihood of a harmful event.

Reliability, on the other hand, has tradeoffs relating to costs. However, in this case, an unreliable subsystem can have a clear safety impact, so its reliability directly feeds into the Hazard calculations. In other words, Safety concerns would dominate.

I hope that actually conveys what I'm trying to say. A bit rushed right now :-)

Maybe they thought they could get away with it, and the magnitude of the problem comes as a surprise?

Short-term PR and benefits: these are much more important than they may seem to be. And also do not forget "keeping my job for these last two years". After all, "we know we do the battery thing right at Boeing, so there can be nothing wrong with them."

Imagine employee A delaying the trials of the "most modern aircraft in the world" because some batteries are "malfunctioning", HE SAYS SO AND IS STILL WORKING FOR US?

It is not just Boeing having a conflict of interests, it is its employees as well.

I haven't accused them of malice, though there may have been malice at some level. But, malice isn't needed for Boeing to have something like this occur. Hubris and a lack of objectivity on the engineering team's part can do it. So can merely getting into too big of a rush.

So you disagree that this is a classical example of conflicting interests?

Is there, however, an individual or small group inside Boeing who got a nice bonus, a pay rise and/or promotions from it?

But it rather defeats the point of having a regulatory body performing safety checks if the manufacturer does those itself, no?

It doesn't defeat the point necessarily, it's just a design point for an allocation of labor that involves various trade-offs. Boeing doesn't just give the FAA a single sheet of paper saying "we've signed off on it, it's all good!" At the same time, the FAA doesn't take delivery of a 787 and do all the tests from scratch by itself. Instead, the FAA specifies process, and Boeing does tests and gives the results to the FAA, and the two sides iterate until the FAA is happy.

In theory, no, because the Boeing employees should be performing the reviews in accordance with FAA guidelines.

If they aren't, that's a problem. But there are so many checks and balances in reviewing aviation engineering work that it seems unlikely that FAA guidelines would get blown off, even if someone at Boeing wanted to.

It actually happens quite a lot in many industries in the US. Not that the government inspectors have a much better track record than the companies themselves. The penalty for screwing it up is particularly high in some areas. I know of one case (food) where the penalties (government onsite workers & other measures) to the operation of a facility applied even after it is was sold to a competitor.

To think that a company will police itself adequately in the face of losing money if rules are enforced correctly is naive in the extreme.

Since when has big business ever cared more about safety than the bottom line?

Normally this would be totally true, but given that every airliner crash is worldwide headlines, safety IS their bottom line.

Safety is always a compromise. The more you spend on safety, the less money you will make. Having the company itself decide when something is "safe enough" is a recipe for disaster, quite literally.

You'd think that would make sense but when deadlines, profit margins and bonuses are on the line, people tend to take shortcuts and sacrifice safty.

Easy example is BP oil spill. You'd think it would be in their best interest to double up on safty because a spill would cost massively more than the safty precautions to prevent it. You'd think. But we see that they take risks and cut corners to meet their deadline while disregarding safty.

What about Challenger? Thiokol engineers wanted to scrub the launch but were overridden by their management over business concerns:

http://www.npr.org/blogs/thetwo-way/2012/02/06/146490064/rem...

It's not exactly the same situation but I think it illustrates the need to have independent oversight from someone outside the same management hierarchy – otherwise it's just too easy to cut corners, particularly since the feedback usually isn't immediate.

It's worse than Challenger. NASA isn't a publicly traded corporation, airlines didn't buy shuttles and people didn't buy tickets on the shuttle. If a 787 crashes due to a design or manufacturing flaw, Boeing stock price will fall significantly, and customer airlines will be very hesitant to order them.

A closer analogy would be the fallout to both McDonnell-Douglas and the FAA after a series of DC-10 crashes. These crashes doomed the company, and they were sold off to Boeing.

NASA isn't a public corporation – although it seems rather unlikely that a disaster getting Congressional attention would be much easier – but Thiokol was a private business and would thus theoretically have a strong incentive not to lose a key contract due to failures in their components.

The problem was that they perceived the risk of perceived flaws as high enough that they chose not to act. Market incentives aren't always as simple as they might seem at first glance.

...after you've eliminated other possibilities like pilot error, poor maintenance, extreme weather, sabotage and so on. When I hear of an aircraft crash, my first instinct is not to suspect the manufacturer.

The same thing could be said about pharmaceutics, but lo, the FDA now exists for a reason.

Key difference: If a drug is found to be dangerous 10 years down the line, all the people who took it have had exposure to whatever risk, long term effects, etc.

If an airliner gets grounded, people who have already flown on it aren't retroactively at risk, so the worst case (other than the one that actually crashes, of course...) is that some airlines have some very large, very expensive, paperweights.

But not their only bottom line.

To Boeing, a 787 with a safety record so terrible that nobody buys it is equivalent to a 787 that ships so late that nobody buys it.

When presented with a choice between risking a terrible safety record and the certainty of shipping late, which do they choose? In a case like this, from a purely financial standpoint, a company would choose to risk safety, and would be completely rational to do so.

The public would prefer a 787 that was a certain failure to one that was a potential deathtrap, but Boeing's incentives aren't necessarily aligned with the public's desires. This is why we have regulation.

All else being equal, Boeing certainly does have an incentive to improve safety, as you point out. But it's far from their only motive, and in many cases it won't be the strongest one.

I actually know a few "big companies" that will enforce safety (sometimes to the extreme - stand on chair = fired) because the consequences of a failure would destroy the bottom line.

The cost of failure in the USA with our courts is pretty extreme. Look at Arthur Andersen LLP (who won its court battle on May 31, 2005) and the effect that had on accounting firms with a consulting arm.

The thing is that Boeing essentially created the FAA and wrote its rules, because nobody else could. And since then they have continued to lead the FAA because Boeing actually makes airplanes. The FAA will never become independent of Boeing unless it becomes an airplane manufacturer itself so it has an independent source of expertise.

Simple. Airbus gets to do the design reviews at Boeing, and vice versa. The FAA is just the referee. :-P

Hey, that's a good idea! We could have a system where two interested parties that want opposite outcomes could present their evidence to a neutral committee, with a government agent determine which arguments can be admitted!

We shall call it "the adversarial system" and test-run it in the legal world!

This is wrong. And that's because safety deals with probabilities, not absolutes. In the absence of absolutes, people who are also responding to contrary pressures (time, money, etc.) can find themselves shifting the lines between likely, unlikely, highly unlikely, and statistically impossible. This is especially true when dealing with complex systems that defy attempts to model interacting forces perfectly.

[EDIT] The 787 is conspicuously (and very expensively) behind schedule. It also represents an especially large investment of ego on the part of senior management, who were basically at war with their unions, and developed the globally distributed production line for the 787 in response to their conflicts with workers in Seattle. This included the construction of a fleet of supersized jets that could move fuselage size parts-in-progress around the world, meaning parts of the 787 assembly line are, quite literally, airborne. In other words, Boeing is no longer operating in the realm of tried-and-true. To the contrary, they're pushing aircraft manufacturing into uncharted territory.

The premise that this was an intelligent response to labor issues has already come under serious fire, in that the distributed assembly line has been credited with an inordinate number of the delays they're suffering from (relying on sub-contracted manufacturers in Italy, for instance, proved especially costly). So there's already a bit of a siege mentality happening at the highest levels of Boeing HQ, and a lot of pressure to pull off what is, in retrospect, looking like a major miscalculation.

Put simply, if they were going to err on the side of caution, they wouldn't have pioneered outsourcing on this scale using their flagship product in the first place. When the "appetite for risk" comes from the top down, it's much harder for subordinates to draw the right lines.

I don't know the details of this specific situation (beyond the article here), but the scenario in general is not uncommon. Employees at avionics companies can receive FAA training to act as internal FAA delegates, reviewing work in line with FAA guidelines, just as an employee of the FAA would do.

Came here to agree with you. Most of the repair stations have in house employees that are designated by the FAA to approve any changes. DER is what it is called specifically. This happens because the FAA does not have the resources to look at every single thing on an aircraft.

If they did, airline ticket prices would be astronomical.

I don't see how the FAA could do differently, given the budget cuts and the rule that prevents them from spending more than 40 hours on a project. Until they can justify splitting up a battery into a multitude of projects (N projects on the anode, N projects on the electrolytes, N projects on the casing, ...), or they are given the means to show their expertise, the checks and balances are engineered for failure from the administration level.

I'm actual surprised its not the NTSB that would do the checks. Its seems like, given the funding, that would be a natural fit and help later in the airplane's life.

It's a bit of a tradeoff. If the FAA fails at some regulatory function, you can't sue the government. If Boeing fails, at least you can sue them.

Who said you can't sue the government?

The government. You can sue them only if they give you permission to do so.

It's called sovereign immunity.

Basically you can only sue the US government over things where it has agreed to let you, like things covered by The Federal Tort Claims Act.

http://en.wikipedia.org/wiki/Sovereign_immunity_in_the_Unite...

http://en.wikipedia.org/wiki/Federal_Tort_Claims_Act

If you got hurt because the FAA failed to do its job wouldn't that be a perfect example of a case where you could successfully sue the government?

Not unless Congress gave you permission to, which they haven't.

IANAL, but I'm pretty sure you're wrong. The Federal Tort Act explains the procedures (and puts some limits on) how you can sue the FAA, but it certainly doesn't prohibit it.

Here's 3 seconds of googling: http://www.firstcoastnews.com/news/local/story.aspx?storyid=... http://www.10news.com/news/faa-settles-lawsuit-with-family-o...

Reply on news.ycombinator.com