Hacker News Reader

Comment by pdonis | Hacker News Reader

While I don't have any sympathy for this pilot, I also find the FAA's excuses here to be less than convincing. Basically, their position is "Well, we trusted this Boeing employee to tell us the truth about the new flight controls, and he didn't." But if the regulators had actually done their job and independently evaluated the new flight controls, they wouldn't have had to take the word of any Boeing employees. I realize that that's the way the FAA does things now, but this whole debacle should be a warning to everyone that that method of regulation is not acceptable. The whole point of independent regulation is to not allow obvious conflicts of interest to harm the public.

View on HN

What you talk about the FAA doing thier own testing and validation is (unfortunately) almost impossible. Back in the 90s, a senior FAA official (I think a Director) had said something along the lines of “The FAA does not and cannot check everything, we just see that companies are doing their tests.” The FAA would require several times more manpower to be able to actually audit or test everything, and that is assuming they have the technical skills (which they haven’t had for highly complex systems and electronics for decades now).

I’m afraid I’m saying a lot of things from memory from the time I had written a report on the Max 8 accidents and an actor analysis. (I’d anyone is interested I could perhaps share it.) Most of this stuff came from the DoT report on the accident, the rest from reputable news articles.

I don't think this style of regulation is limited to FAA. FDA does not independently test all drugs either. They just review and ensure the drug companies' tests are acceptable.

And FINRA is made up of brokers and banks but overseen by the SEC. The EPA doesn't directly monitor every factory. The NTSB sets general testing guidelines and standards and seems to have greatly increased auto and aviation safety, but they can't police everything and at least to the public seem to regulate after the fact.

The general term, I believe, is self-regulatory-organization. The theory, I guess, is that the government sets the laws and says very generally, "no fraud", and "you have to write your own rules and make sure they're good", but offers little technical guidance otherwise. I think this could work well if there were very heavy penalties for failures to self-police, but in practice the revolving door between government and industry incentivizes slap on the wrist style punishment. It's a hard problem and I don't think we're doing well over the last few decades.

... and for most of the agencies, it's not that their budget is small or the agency itself is incompetent: it's what's required by law and they can't override it unless Congress decides to change the laws.

You don't want regulatory agencies creating law on their own though. Not only is that unconstitutional but the reason its unconstitutional is because you would inevitably end up with some busybody who is not accountable to the people massively over stepping their power and creating tyranny.

Regulatory agencies create "laws" all the time. When Congress creates legislation, they rarely spell out all the details and implementations. This is left to the agency to interpret, and they have a wide latitude in both interpretation and enforcement.

You're referring to Chevron deference and/or Auer deference, and it is worth noting that those are in the long process of being curtailed/attacked heavily. :(

Chevron defense is separate and distinct from Auer defense. The Chevron defense applies to Congressional legislation, Auer is applied to an agency's own vague regulations. I have no issue with restricting the Auer defense. If an agency issues vague regulations, then that's their own fault. You're correct that they're both being attacked, but the underlying issue won't go away; Congress can't/won't dictate every detail of legislation especially when it comes to general/vague legislation. They have neither the inclination nor domain knowledge to do so.

> Not only is that unconstitutional

While I do study relevant US laws as required in my job (requiring familiarity with how law operates in countries where we operate), I am not an American, but I do understand certain things with regards to the US constitution and relevant case laws.

The Congress originally meets only for a few months' time, usually less than 6 months. This is due to the reality of the time, where travel is slow and representatives only receive a comparable salary to most people. Thus, it is exactly empowered to delegate certain powers to the executive branch. As someone mentioned, the administrative law is a cornerstone law and yet it delegates many powers to the executive branch.

In fact, said law and many, many, many similar (federal) laws have been upheld constitutional in the Supreme Court. There are certain powers that only Congress can do, and cannot be delegated to the executive branch, but it is clearly laid out in the constitution what those are (notably spending). Now I said federal because in certain states, the legislature can only delegate in very narrow situations (usually only in cases where lives would be in danger or in the protection of properties and where a need of immediate response is demonstrated).

So I'm confused why are you saying that is unconstitutional, in fact American history shows a very different answer. If you think that should be not allowed, you're entitled to your own opinion. However unless I read it incorrectly, the constitution, even considering the various amendments, is unfortunately not aligned with your opinion.

> the constitution, even considering the various amendments, is unfortunately not aligned with your opinion.

Sure it is. Article I of the Constitution says that all legislative power shall be vested in Congress. That means anything that has the force of law--and all Federal regulations created by executive branch agencies under the current US regulatory regime have the force of law; you can be fined or jailed for violating them--has to be passed by Congress using the process described in Article I. So any Federal regulation that has not been passed by that process--i.e., every one of them--is unconstitutional.

The fact that current US jurisprudence disagrees with that statement just illustrates how far current US jurisprudence has diverged from what the Constitution actually says. The status of Federal regulations is by no means the only example: current US jurisprudence says that Congress can regulate farmers growing crops for their own personal use because of the Commerce Clause; and that a city government can use the eminent domain power to evict people from their homes and turn the property over to a private development corporation (that ends up never developing the land anyway), and that counts as a "public use" under the Fifth Amendment.

> You don't want regulatory agencies creating law on their own though.

While I agree with this as a matter of personal opinion, it is not at all the actual fact in our current regulatory regime. Federal regulatory agencies create law all the time. Look at the Federal Register; every regulation in there has the force of law and was written by a regulatory agency.

> you would inevitably end up with some busybody who is not accountable to the people massively over stepping their power and creating tyranny.

Which is exactly the situation now.

Look up Administrative law, and prepare for the pucker moment when you realize that what you described is actually exactly how it works.

It's called rule making authority. Normal, necessary, obviously correct form of governance.

I'm afraid this is the logical consequence of a majority of people falling for the lie that "government can't do things as efficiently as the private sector".

When people vote for politicians who say

  "government is wasteful",
  "there's too much red tape"

what did they think was going to happen?

I don't think so, testing like you refer to is not scalable and even more inefficient.

Letting an external auditors understand all the small technical details and test them independently will basically halt any progress. It might make this specific change safer but for the long run will slow down innovation and development of better and safer products.

This has some similarity to software development- we test much better but prefer to move faster to achieve a better overall quality and be able to fix issues faster and better

When politicians say "government is wasteful" or "government can't do things as efficiently as the private sector", what they're saying is that their management style is wasteful.

Would somebody asserting that about _their_ _own_ _job_ even make it past the interview in the private sector?

Or they could be saying that centralized monopolies are rarely efficient. That is certainly my experience. For example, SpaceX undercut NASA by a factor of about 10, according to this FT report: https://www.ft.com/content/25e2292b-a910-41c8-9c55-09096895f...

plane-pal aircraft coming soon to a gate near you!

NASA sources their materials from the private sector. The private sector is notoriously "price gougey" when it comes to government money.

Government actors are responsible for acquiring material at normative costs. Their failure to do so is their own.

Normal costs. https://en.wikipedia.org/wiki/Hypercorrection

"Price gougey" implies that the prices are unethical; not in compliance with ethical norms; not normative.

'Not normative' doesn't mean 'not in compliance with ethical norms'. It means not pertaining to ethical norms. It's not the correct word, however you twist it.

Here's some more info if it helps: https://en.wikipedia.org/wiki/Normative_statement

Exactly! And when all US companies refuse to lower their prices, they should start sourcing from foreign companies to increase competition!

Oh hey look, I'm signing yet another contract with an arbitration clause when signing up for a cell phone service. Must be my fault that all of them require the arbitration clause...

What they're actually saying is "If you sponsor us we will make sure government money goes your way."

Efficiency - or the lack of it - is absolutely not the point.

I think the worlds greatest agency couldn't evaluate a so complex system as the Max. They should just have said no.

I believe it is foremost a complexity issue, and the MCAS(?) failure is just one of the many things that could go wrong, that actually did. Boing probably have more issues with the plane that is waiting to surface given their culture ...

Saying "no" would have been an evaluation.

It's a management function. If someone comes to you and says "We want to rewrite everything so it runs on a Raspberry PI powered by a hamster wheel" you don't need to ask about the engineering spec of the hamster wheel.

The Max MCAS was only marginally more plausible in overview. Details were never going to rescue it.

Indeed. The FDA just has a multi-step process where you need to define how you'll run your trials, what data you'll collect and whether it's enough to get approval.

The FDA very carefully reviews the submissions, requires validation of tests and safeguards so data can't easily be manipulated. But the system is built on trust. If a company wants to manipulate, fake or exclude negative data they can. They'll likely get caught, but not always.

Yeah, people are are drastically underestimating how enormous the testing apparatus is at a company like Boeing. It's thousands of employees. Expecting the FAA to duplicate those efforts is ridiculous.

I don't know much about the FAA, but as someone who works in healthcare, I know the FDA conducts regular audits of medical device manufacturers.

They roll in for a week, request access to everything and everywhere, then pick a handful of areas (randomly) to do a full deep-dive. Generally, if a company is cutting corners, discrepancies will exist in many areas and they'll quickly spot one or more of them. I assume the FAA operates similarly.

And ISTR that just around the time I entered the medical device industry, that this is exactly what happened to Abbott and they were forced to remove some of their products from the market for months until they could bring their quality system back up to FDA satisfaction.

Hard to believe that it was so long ago. Man, I'm old :-) https://www.chicagotribune.com/news/ct-xpm-1999-11-03-991103...

Similar for many other things, the report most countries based the safety of roundup on was written by Monsanto itself. Most countries only check these reports for completeness and obvious errors, they don't try to run the studies themselves. However that "error checking" of the report is something where the FAA fucked up, the information on the MCAS provided by Boeing was completely out of date, either the documented testing procedures where incomplete or the FAA did not notice that the MCAS handled cases far outside of its original specification.

Of course the FDA has to regulate 10s of 1000s of companies\drugs\trials. The FAA basically just oversees Boeing and Airbus in the large commercial jet market and either offers 10s of models...

The FAA oversees every flying object in the US. Not just new models, but also config changes to existing models, continued airworthiness, all the design org and production org certification of manufacturers (per site) and suppliers (again, per site). Which is quite a workload, especially in a world as complex as aerospace.

The point remains.

What point? That the FAA screwed up? That Boeing intentionally lied? Those were nver in question.

If it is the point that the FAAs job should be easy because the only oversee Boeing and Airbus, well that point is just ignorant.

If you're unable to understand LattleLazy's point it's probably best to refrain from responding to them until you do.

Attacking the poster is not moving the discussion forward!

That there's multiple other airplane manufacturers, not targeting the "large passenger jet" market, but more towards the "general aviation" market?

Textron (Cessna, Beechcraft, hawker), Piper, Carlson, Viking, ... There's dozens of aircraft manufacturers (I have intentionally not listed those who make hit planes, but I think they too fall under FAA checks).

I would not actually be surprised if the small airplane manufacturers puts substantially higher load on FAA than Airbus and Boeing do.

The FAA oversees all portions of pilot licensing, airplane manufacture, airplane service, operations, traffic control, idiots with drones, etc, for general and commercial aviation in the entire country.

Certifying new commercial models is but a small portion of what they do. By number of employees and budget I would bet that ATC is actually their biggest responsibility. That's not to say that they should just rubber stamp all that...

Your comment boils down to a false dilemma fallacy between "FAA blindly trusts manufacturers" and "FAA conducts their own testing and validation." Clearly they could have been more actively reviewing stuff coming from Boeing, or engaging in some level of auditing.

If the FAA had paid attention they would have seen a company desperate to compete modifying numerous basic characteristics of an airplane to the point of making it aerodynamically unstable, using a flight control system as a bandaid to fix this.

Not to mention being such massively cheap assholes that they literally didn't install warning lights in the cockpit to tell the pilots when the sensor their flight computer would use to override control inputs, had failed.

Actually, this is a sauce effect of the complexity of modern systems: regulators have no choice but to trust companies and regulate the results of tests, with heavy penalties for cheating. Consider pharmaceutical trials: it's impractical for the FDA to check on each patient. Financial audits often assume the client isn't directly lying to the auditor, with criminal penalties as the disincentive.

The alternative requires raising costs and slowing progress and innovation, which is a nonstart in a competitive environment such as Boeing vs Airbus.

It can be done, it just costs money. It can not be done within the liberal frame of mind, were everything that costs money is tax and thus theft.

So why not reformulate it correctly. In my ideology, it is not possible to solve this problem.

If that ideology would be out of the way, it could actually be tested pretty good via unittest running a simulation. So once developed those tests would actually be pretty cheap to run.

Its just this ideologic blindspot that prevents good safety.

> It can be done, it just costs money.

Well money and time.

You speak as if you think that if only the great billionaires accepted a little less money then we could have safety.

How many lives are you willing to lose for a multi year delay to accommodate the FDA recreating every required test that the pharmaceutical company did? How many lives are currently lost because the current system doesn’t work?

Your suggestion is akin to voter id’s as a requirement to prevent voter fraud. How many legitimate voters are going to be prevented from voting due to the new rules to stop how many prior confirmed cases of voter fraud?

This isn’t a lack of resources. This is society deciding that the resources are better used elsewhere.

> This is society* deciding that the resources are better used elsewhere

* Actual decisions on resource allocation made by regulatory agencies, which have a close partnership and history of employment with the industry being regulated.

It's different to say "society decided" vs 'a specialized subset of society, with tangled incentives, decided.'

At the end of the day, it's a spectrum from (no oversight) to (full, independent validation).

Boeing didn't want MCAS highlighted as a change, Boeing didn't want the FAA to independently discover it, and Boeing got all these things. Either by action on its part or by design of what the FAA did and did not independently verify.

That's a strong indicator we should shift regulatory posture further towards (full, independent verification). And while it may be cost prohibitive to shift all the way there, that's doesn't mean we can't shift closer to it.

This is the only real reasonable stance in light of what happened with the 737 MAX.

Did society decide this distribution of resources? Or was it lobbying?

Our society, as presently organized, delegates that choice to lobbyists. That is one failing.

The problem with the FAA vs Boeing is not that the system is designed around the (correct!) assumption that the public's and FAA's interests, on one hand, and Boeing's interests on the other are aligned. This failure cost Boeing enormously!

What was not aligned were Boeing's interests and Boeing upper management interests. The pervasive failure of our society to force CxOs to align with the companies they run and with society at large is much bigger than just in aviation.

That some pilot was indicted, but not the management he was responsive to, is a glaring indicator of this failing.

But in the case of aircraft we pay for that innovation and "progress" with dead bodies. Maybe it is time to rethink the cost/benefit of how we enforce these regulations.

LOL speling: sauce effect => side effect!

The 737 Max is not aerodynamically unstable. I don’t know where people got this idea. The purpose of MCAS is to match existing stick pressure progression in certain situations. It’s not a fly-by-wire system like the F-16.

Not just existing stick pressure progression, but required stick pressure progression. As far as I understand it, in that specific situation the 737 MAX without MCAS does not comply with the rules regarding aerodynamic stability. It's not dramatic, doesn't manifest itself in normal flight, and could be dealt with by pilots quite easily as far as I understand it. But it's a rule, and it was important enough for Boeing to first put a quite gentle MCAS on the plane, and then to increase the effect of MCAS.

This is correct. It is non-compliance for control stick forces to slacken towards a stall on a passenger carrying aircraft. They put a gadget in the system to render it compliant.

Unfortunately, they did not work as hard as they should have to ensure that it would not malfunction. Or to make sure if it did pilot's were aware and had a chance to develop muscle memory for it.

All of that was Management's push.

I've heard people break the design/implement/test phases down to essentially equal parts in terms of cost. It's not a small task if you think about a government agency's testing capabilities potentially needing to be roughly 1/3 of what the private industry is throwing at it. I can see where they need to be pragmatic and adopt a stance where they're really just overseeing the vendor's testing and doing some spot checks. Not ideal, though.

It wouldn't require several times more manpower for the FAA to figure out there's something called MCAS that didn't exist before, would it? (Note that I'm talking in the current world we live in, not in a hypothetical world where Boeing would be a mortal enemy of the FAA doing everything within the stretch of human imagination to hide MCAS.)

Out of curiosity, how confident are you that there aren’t any other novel systems that didn’t exist before on a similar level as MCAS?

Having not had a public debacle around them, how much effort would it take for you to personally certify that MCAS is the only novel system of its caliber on these aircraft?

The airplane couldn't have been built without many system diagrams showing MCAS, or a similarly complex component.

The FAA should have pulled schematics directly from Boeing engineering during certification. And the FAA should have someone with enough technical expertise and experience look at them. And that person should have said "The submitted information by Boeing doesn't include full details on this subsystem."

Whether or not the Boeing test pilot highlighted the system for the FAA is a red herring. It's the FAA's job to find this, regardless of whether someone points them at it.

If the FAA doesn't have the technical staffing or expertise to do this, then that's the problem. Charging the test pilot is necessary, but not sufficient.

If the FAA infrequently performs this work (certification of a new aircraft), then flex in retired expertise! You can't tell me there aren't qualified, retired candidates (ex-industry or ex-FAA) who would have signed up for a year or two review. And all the better that they don't have career incentives!

Essentially, this is the FAA charging Boeing for not doing the FAA's job correctly.

Now you're talking about the FAA essentially replicating the scope of Boeing's test/QA operation. There are thousands of pages of diagrams and likely millions of lines of code. One person is not going to scan them and say "Here! This is a problem."

> Out of curiosity, how confident are you that there aren’t any other novel systems that didn’t exist before on a similar level as MCAS?

Me? What makes you think I would have information on this?

If I had to hazard a random guess, I would think that, if you're talking about the MAX 8, there have been enough leaks and testimonies and whistleblowers that any comparable system would have probably been mentioned somewhere. I have no idea either way, I haven't read everything that's gotten out. But I don't see why a regulator couldn't use various means to figure stuff like this out with reasonably high confidence.

It would be nice if the FAA were able to bill large companies making planes the public is intended to fly on commercially for all of the hours required to process the 'type certification' fully and exhaustively.

That process would include validating all aspects of the mechanical specifications and changes of parts (reused already OK parts from the same authorized suppliers would be a quick check-off), mechanical engineering, electrical engineering, computer software, and any changes for maintenance and end operators.

Describing it fully like that, I believe the only benefit to 'type certification' should be training for the end users, but major overhauls should require retraining and that should be caught.

From the article: >Because of his alleged deceit, the FAA AEG deleted all reference to MCAS from the final version of the 737 MAX FSB Report published in July 2017

The FAA was aware that this system existed.

AFAICT (medium certainty), the development sequence went thusly: (1) MCAS added to design, (2) FAA informed about MCAS, (3) MCAS potential control inputs and overrides drastically increased during development, (4) FAA not informed about changes, (5) FAA certifies aircraft on basis of (2).

So it's probably most accurate to say "the FAA was aware of the system existing, but incorrect on the details of that system."

Like, how many new major passenger planes do they have to test every year? 0.1 on average?

Let's say it is 0.1 on average, and the FAA would need 5 years just to test. That gives you an expected wait time of 10 years to get something tested, and 20 years if you include development time. I'm sure the industry benefits from lower turnaround times than that.

I would imagine they could test multiple planes at the same time.

Sure. So you take half the staff and put them on testing a second plane. Then each individual plane takes 10 years to test instead. (Remember that you don't gain person-hours for free just by shuffling people around.)

What would this accomplish? Instead of a total system time average of 10 years, you now get an average testing time of 10 years to which you still need to add the queuing time, so you're even worse off than before.

(The queueing time won't be five years with two servers in parallel, and I can't do the exact approximations in my head, but it'll be at least two years. In other words, by testing in parallel you worsen the cycle time from 20 years to at least 22 years.)

This is a good general rule: by taking on more work in parallel, you'll make the turnaround time worse. This is why lean consultants go on about limiting work-in-progress.

Also a call to learn some basic queuing theory! It comes in handy often.

... or increase the staffing[1] to a degree where you can do more than the absolute bare minimum when lives are on the line

1: https://www.aviationtoday.com/2021/06/03/faa-asks-budget-inc....

When there's a defined process that appears to run in isolation, I don't see why there should be only one queue in this case, considering that the task length cannot be easily reduced.

I'm all for increasing the FAA budget so they can do their jobs better. I think that's a net positive for the industry.

What I was saying with my previous comments was that giving the current budget levels, it doesn't help to shuffle people around (without some strong assumptions on the process, which in my experience rarely are true in practise.)

Only if the workload doesn't shake out favorably in light of Amdahl's law.

Most mechanical processes aren't necessarily conducive to parallelization. Verification and information processing on the other hand can do favorably in the presence of non-reliance on a physical system-under-test.

Which subsystem vetting arguably is. If you're talking vetting specs.

Yes, there are some circumstances under which it makes sense to parallelise some of the work.

I'm assuming the FAA does this already. Human organisations have a bias to parallelise to a fault. Increasing the parallelism level beyond this does not improve lead times.

Or, more precisely: How many new major passenger planes a year that are trying to imitate the same type rating as an older aircraft do they have to test a year?

In that specific case probably the important parts to review are the bits that try to make the new airframe handle the same as the old one.

None of those are unfixable problems at the FAA though. It all just boils down to a hiring problem. (If there were a will to actually have a functioning government.)

> What you talk about the FAA doing thier own testing and validation is (unfortunately) almost impossible.

No, it isn't. It's just more work than the government feels like doing, involving more technical skill than the government feels like hiring.

The problem is that the government can't have it both ways. It can't both claim that it is regulating airlines and airplane manufacturers to protect public safety, and also claim that it can't independently check what the regulated entities are telling it. It has to be one or the other: either we get the actual independent regulation that the government claims to be doing, with whatever resources it takes, or we all admit that we are not going to get that because the government is incapable of doing it, and we figure out some other way of ensuring safety.

That sounds like they are knowingly incompetent? If a utility regulator had no one on staff who knew how utility scale power transmission/generation/whatever worked, that’s what we’d call it for sure.

And aerospace engineers are dirt cheap. What sort of clown show is the FAA running?

> That sounds like they are knowingly incompetent?

That sounds like they are knowingly underfunded, which they are.

> And aerospace engineers are dirt cheap.

They are? Where are all the airline startups? I assume they measure in the hundreds with all this dirt cheap talent running around.

Aerospace engineers have as much to do with airlines as software engineers have to do with data entry companies. Which is to say pretty much nothing.

Aerospace requires lots of capital expense and is very ‘large customer’ driven and aerospace engineers play second (or third) fiddle to that, unlike in software.

maybe this industry isn't like software where all equipment you need for almost all jobs is 16gb ram, nvme m2 disk, decent cpu laptop

> And aerospace engineers are dirt cheap.

I suspect you're letting your software world experience dictate your expectations of what engineering is all about, and what it takes to actually get work done.

Software development is a rare field where the only relevant resource is man hours. In other fields, including aerospace engineering, trained meat bags tend to have a negligible cost to the point where replacing a whole engineering team might be a minor inconvenience. However, crashing a prototype is a project killer due to cost alone.

It sounds a whole lot like you’re agreeing with me - but seem to think you’re not?

I believe the OP's point is that if you're not just using engineers to audit the tests Boeing does, then additional engineers are a fraction of the cost of running a full parallel aerospace test programme. The capex to set up the test infrastructure is significant even if Boeing is legally obliged to supply you with prototypes to destroy at no cost (which obviously has an impact on the amount of R&D Boeing is willing to do).

Also, you're not hiring from a diversely employed Valley pool: most of the engineers with the requisite level of understanding to test Boeing's hardware work for Boeing (and to an extent its supply chain), which might mean you don't have to offer them much of a pay rise, but it also means [i] you're weakening the engineering capability of the firms actually designing and building the stuff by poaching them and [ii] their views on what's safe and what's an appropriate level of testing aren't fully independent anyway.

Eh, that doesn’t sound like it?

1) Boeing lays off thousands of aerospace engineers regularly (they did as part of the max disaster), and doesn’t rehire them all back - the industry is highly cyclical, and Boeing is periodically shifting locations anyway

2) the stated concern was Boeing brass was applying undue influence to engineering correct? If those folks worked for the FAA directly after being laid off , wouldn’t they be more than happy to stick it to Boeing brass if they were telling them to cut corners?

3) we’re talking design overview and identifying where Boeing (or others) may be ‘putting their finger on the scale’ or trying to snow regulators by asserting bogus test results or designing tests that they can pass by not including important test criteria they may not pass right? That is certainly something an engineer who was previously in the industry would be aware of, or even a independent engineer should be capable of spotting from ‘the outside’ - and require they do.

4) at (linked in a parallel thread) a median salary of $118k, which is well within something the feds could cover, the FAA can certainly afford to hire a non-token amount of aerospace engineers onto their staff if they actually wanted too/Congress wasn’t trying to kill them. This isn’t like hiring on a FAANG staff software eng for 700k or whatever which would cause outrage or break the pay scale, and this is for something for which there are clear large body counts that can be pointed at.

Now if we want to say Congress has been strangling the FAA for a long time (like the IRS and USPS) and forcing them to outsource to industry or whatever, hey - I could believe it - but that is something that should be yelled from the rooftops because that can be fixed, and that will cost us a lot in blood.

I don’t want more Americans dead due to corruption of a regulatory process, especially not my friends or family, and those are the stakes here.

> Eh, that doesn’t sound like it?

It should, as it was what I said. I'm really not sure if it's possible to make a point any clearer.

> Boeing lays off thousands of aerospace engineers regularly (...)

Sounds in line with the classical big corp style of management. I'm not sure what any of that has to do with humans not being the critical element of providing a service. In fact, are you sure you're not supporting the point you're trying to refute?

You said that the FAA would be causing brain drain - in a field where thousands get laid off all the time?

And that Boeing would surely be exerting influence on them so they wouldn’t catch issues - after Boeing laid them off?

And that it wouldn’t matter having competent engineers at the FAA because catching things require expensive tests - that the engineers if they existed at the FAA could mandate Boeing pay for, since they would know they needed them to do them?

Huh?

No one is saying the FAA should be running a full parallel aerospace program. I’m saying if they lack in house competency to call bullshit on what a player they are regulating is passing to them and relying on that player to just always do the right thing, then they are not effective regulators

It would be like taking Facebooks word that they are totally being good privacy wise, and not having anyone available who understands internet tracking or adtech. Which, is of course another failing regulator (looking at you FTC), but at least that doesn’t get hundreds of people killed in giant fireballs?

I think you’re deeply misunderstanding the purpose of the FAA and the magnitude of the problem here.

Like almost every other agency (the FDA didn’t do the vaccine trials; are they incompetent too?) they don’t do the tests. They just police the industry.

Theoretically if you set laws and regulations and dole out severe punishment for bad behaviour, you don’t need to be the one running the tests.

People should want their government to run on trust (if their culture is compatible with trust) because it’s far cheaper and more efficient.

'and dole out severe punishment for bad behaviour'

When will Boeing CEO and Board of Directors get locked up as part of the severe punishment?

Presumably when it is shown beyond a reasonable doubt that they conspired or deliberately overlooked the fraud. The prosecutor has every incentive to get participants to roll over and follow the conspiracy as high up the ladder as it goes. Successfully prosecuting such a large profile case, particularly with defendants with very little public sympathy, would make his/her career.

>The prosecutor has every incentive to get participants to roll over and follow the conspiracy as high up the ladder as it goes.

This is naivette.

See this:

https://www.corporatecrimereporter.com/news/200/lead-boeing-...

From the article:

the case was settled with a deferred prosecution agreement — an agreement that Columbia Law Professor John Coffee at the time called — “one of the worst deferred prosecution agreements I have seen.”

Boeing did not have to plead guilty to any of the allegations.

No Boeing executive was charged.

And the Boeing deferred prosecution agreement included an unusual provision finding that a compliance monitor was not necessary because “the misconduct was neither pervasive across the organization, nor undertaken by a large number of employees, nor facilitated by senior mismanagement.”

“That is without precedent,” Coffee told Corporate Crime Reporter earlier this year. “I have not seen that anywhere else and I’ve looked at a number of deferred prosecution agreements. Prosecutors themselves are not conducting the investigation.”

Boeing’s lead corporate criminal defense law firm is Kirkland & Ellis.

Erin Nealy Cox, the lead prosecutor in the Boeing case, left the Justice Department earlier this year.

And last month she joined Kirkland & Ellis as a partner in its Dallas office.

Wow. Thanks for the background. It seems to me there should be a special investigator appointed to look into potential prossecutorial misconduct here.

Since you’re not going to find any documentation of that because of CYA, and Boeing is the flagship US aircraft manufacturer with a protected (as in ‘in the interest of national security’ protected) position to offset Airbus - that’s pretty much not going to happen.

I think it would be possible to ring-fence the individuals to be prosecuted, facilitate an orderly handover of power inside Boeing, and then prosecute the individuals without risking mortal damage to the company.

That ring should probably extend around McDonnell Douglas' management from the 90s, which made their ways into the upper echelon of Boeing during the merger.

So if the FDA is unable to find someone who can understand or can’t interpret/understand the studies themselves enough to see flaws or likely fake data, they should just take the companies word for it that it’s all good?

Last I remember this being a topic of discussion, the stance was ‘trust but verify’ no?

While it may be efficient to rubber stamp things, it is not doing their job. Folks scam all the time, especially if they know no one is looking.

If they lack the competence to be able to independently verify, they aren’t being effective regulators.

> If they lack the competence to be able to independently verify, they aren’t being effective regulators.

Bingo. If I use 3 managers and PMs who don't know how to program for a code review, then that's not a very effective code review.

> And aerospace engineers are dirt cheap

Inaccurate, please stop sharing misinformation. Seeing this on HN is unfortunate.

Do you have any data to share? Most of my family has been involved in Aerospace, with my dad working for skunkworks, my brother working for Garmin (and a formal aerospace engineer).

Most aerospace engineers are lucky to break 75k/yr to start with little to no equity, and often need to move to the middle of nowhere (compared to say NYC, SF, LA, etc for software), and get hit with periodic catastrophic layoffs with the regular cycles in the industry.

It’s pretty common that software folks are paid 2-5x with far less intense or zero credentialing and better work conditions - at the same company.

https://online-engineering.case.edu/blog/highest-paying-engi...

Median numbers, so masks variability. But it looks like AEs are generally paid decently, relative to other engineers.

https://www.bls.gov/ooh/architecture-and-engineering/mobile/...

Oof, even worse than I noted. So median (and that includes established mid and late career aerospace engineers too) is $118k all in?

That’s roughly half of the initial comp for an entry level software engineer at any of the SV firms, and most folks will be making much more than that at said SV firms within a couple years.

Being able to get a team of 4-5 experienced and credentialed aerospace engineers for the comp of a single ‘senior’ (mid-level somewhat competent but not amazing) software engineer sounds dirt cheap to me?

You realize you just compared Silicon Valley software engineer salaries in high margin industries to all location aerospace engineer salaries in normal margin industries?

If you want to baseline off SV salaries, you shouldn't be looking at median all-AE numbers.

You’re the one that provided the numbers?

We’re on a SV startup website, where the comparison to cheap or not is of course going to be based on this.

You provided as a counterpoint to my statement on aerospace eng’s being cheap, data which shows median salary across all experience levels of the field being half the starting pay of a typical entry level SV software engineer - which typically requires no specific credentials, unlike Aerospace engineering.

If there is a large cluster of companies who pay 4x the median aerospace engineer salary to Noobs, then please provide said data. My understanding is those don’t exist.

SpaceX, a high profile name and maybe the closest to a SV type place you’ll get in the industry pays between $70-100k to their Aerospace engineers, based on multiple sites. Here happens to be a random Reddit thread about it in the first couple results.

[https://www.google.com/amp/s/amp.reddit.com/r/engineering/co...]

Which is exactly the point I’m making. When a straight out of school software engineer has a whole section of an industry they can go to that will pay them 2-5x what an experienced aerospace engineer mid-point or even late in their career can make ANYWHERE (except MAYBE a one-off consulting gig somewhere), then aerospace engineers are cheap no?

FAA could facilitate smoketesting and focus on parts they deem the most suspicious/experimental/new in a new airplane. And then outsource the actual tests to a third party.

By law, the certifiction activities done by the EASA (I assume the FAA is the same) cannot be done by third parties. It also worth noting, that the aircraft certification procedures worked pretty well for decades, up until Boeing started to lie to the FAA. Not sure how Boeing kept its design org approval after that.

EDIT: Some required activities by the organizations to be certified can be outsourced to third parties, the org itself is still accountable.

> Not sure how Boeing kept its design org approval after that

That's an easy one. It's too big to fail. One way or another Boeing had to make it out the other end of the 737 Max disaster intact as an organisation. Anything else would have been unpalatable from the point of view of the American military industrial complex. I know that phrase is usually applied in a derogatory way but here I don't even disagree with the thinking.

That seems to be the answer. And I kind of get the reasoning. Under EASE rules, there are accountable people at the head of approved design and production orgs. I hope those people at least got their licenses revoked, if something similar exists under FAA rules.

What could possibly go wrong?

What manpower would be needed? How often are new aircraft validated? That's pretty scary that at the end of the day it's just a corporate shill pilot validating. What is the point of all the regulations if that's all that it takes?

Every single component in every single aircraft certificated in the USA is tracked all the way to the mine. A quick web search reveals around 11.3 million people work in the aviation industry directly. A significant portion of this would have to be duplicated if the FAA were to verify everything.

Perhaps the FAA could get Airbus to contribute a review of the test plan when a new plane is close to release. Then you'd certainly see a more lively technical debate.

A pair of eyeballs, a brain and a set of teeth are all a regulator needs, in other words.

> would require several times more manpower

Or they could be several times more efficient. $17.5 billion/yr can do a lot.

Also this feels a lot like Boeing offering up a sacrificial lamb. I find it hard to believe that the FAA can focus on one person at Boeing as the reason for the failure in reporting the issue. There are too many managers and engineers involved. On the surface this sounds a lot like the Challenger disaster story.

Overall though, I'm at least happy some individuals are getting prosecuted, it sends a message that there are legal risks to individuals involved in this kind of thing, not just diluted corporate responsibility.

What really needs to change is the corporate culture that led a person to think that they would be doing the company a favor by lying to the regulators. This probably goes all the way to the top.

Also, whomever decided that a basic software safety check would be an optional extra with a price tag should definitely not be in the management chain. That's the sort of next quarter profit-only thinking that rots companies from the middle out. That is the kind of thinking that results in your brand new product killing 346 human beings.

The "software safety check" wasn't an optional extra. It didn't exist at all! The optional add on was for an "AOA DISAGREE" warning light, which would have indicated that the MCAS might be relying on faulty data (if the pilot was even aware of MCAS in the first place) but wouldn't have actually stopped it from doing so.

Now, the updated MCAS will only activate if both AOA sensors agree. Which seems like a fucking no brainer that should have been the case from the start, but... yeah.

Only if the people actually responsible get prosecuted; prosecuting the wrong person sneds the message that the people really responsible get away with it - so there is need for some care.

Even prosecuting the wrong people at least encourages whistle blowing. If someone thinks they might be the sacrificial lamb if excrement hits the fan they can protect themselves by being the first to report it to regulators

Is Boeing offering a lamb, or the justice dept applying pressure to get him to cooperate and try to get higher level targets?

At least it is somebody with a big title. Chief Technical Pilot.

Genuinely curious; how big is that title? Are they essentially an overall project manager who gives the green light? Do they have the power to tell the executive class "Nope" without getting fired?

Never mind. I saw Buildsjets comment that explains his position.

I think the other approach is to have more corporate whistle blower programs with teeth and upside. In a case like this, lots of people knew everything wasn't on the up and up. But it is hard to get someone to understand something when their job depends on not understanding it (Upton Sinclair). However, if the whistle blower programs came with immediate cash (after initial proof was obtained) with more to come and potentially relocation and identity change, people would be more likely to come forward.

Fighting something like this in court plus media scrutiny will basically ruin someone's life and make them almost unhire-able in their field. Excepting someone to ruin their family's life for the greater good isn't likely. There won't always be a young single ideologue who is willing to move overseas to escape his own government. There are lots of major crime systems where someone is the spouse of the criminal and knows what is going on but how are they going to give up their home and life for their kids while also putting their life at risk for the sake of doing the right thing. People have shown that they will do the right thing if you make it easy enough and safe enough.

I can’t help but feel that this is all by design. Legislators require financing from corporations for re-election, corps don’t want whistleblowers because it risks their bottom line. Therefore legislators don’t improve protections.

Does everything come back to campaign finance reform?

It could be a factor for sure. I don't think campaign finance reform will ever happen but whistle blower could. I've given up hope on the altruistic billionaire to do things the government is reluctant to.

> basically ruin someone's life and make them almost unhire-able in their field.

It will make them almost unhireable everywhere. Even outside their field, they're still a risk. They've now got a very public "troublemaker" label.

The prizes would also have to be pretty large. We're talking "never have to work again" large, because that's a very real possibility, or at least a very real perceived concern.

We're talking people who make a good living. For people not close to retirement, it's probably a multi-million dollar number. At $200k/year, that's $2 million per decade, and not including potential raises. After you pay income taxes on the prize, you probably need it to be close to $10m to break even.

I was assuming they would be even higher. However, look at the economic damage being done by things like the opioid epidemic or the lives lost from these lies inside Boeing. We spend hundreds of billions on boondoggles, it doesn't seem like putting aside $5B per year to make the world better is much of a strain for the US government.

How were they supposed to do that? They'd have had to either review the source code and somehow notice that it had too much control authority over the horizontal stabilizer, or run the flight control system in a simulator that reproduced the failure conditions found in the field. These seem _possible_ but not exactly easy. It'd have been much easier for the Boeing engineers who designed the thing so badly to have had bosses who said "wait...what? No you can't do it like that".

As I understand it, the source is a very careful implementation of a spec, and the FAA should have the spec.

In any case, I strongly doubt that the buck should really stop at this test pilot. Someone higher up surely has some degree of responsibility. The test pilot did not invent MCAS.

> It'd have been much easier for the Boeing engineers who designed the thing so badly to have had bosses who said "wait...what? No you can't do it like that".

Interesting. I never even imagine that happening anymore. I find the reverse so much more plausible: the bosses say to design it like that but the engineers have the backbone to say no (even at the risk of being fired and replaced).

On the one hand, I see where you're coming from. Executives are the ones with the money and power, so they should have the responsibility. They should go to prison when they order misconduct (whether that's negligence, fraud, etc.). This should incentivize them to act conservatively. (Where right now, we reward them largely based on short-term stock performance, and we never punish them, and I think our executives are overwhelmingly sociopaths who pursue short-term stock performance above all else.)

On the other hand, engineers are the ones who throughly understand the issues and have licenses with ethical standards attached. They're going to notice the problem. I believe many already aren't willing to sign off on something they don't believe is safe. With good enough whistleblower protection, they'd be likely to speak up when they see someone else signing off improperly.

FAA could have looked at how the fans were upsized and pushed forward to avoid dragging on the runway, and called for a new type certification. The FAA failed by allowing Boeing to not get a new type certification, which would have required greater scrutiny of the new airframe.

the airframe was fine. the problem was that mcas was implemented in a totally shitty way. If it had just replied on 3 sensors instead of 1, wet likely wouldn't be talking about it now.

No, the airframe was different and didn't handle the same way as the old one. In order to hide that, Boeing implemented in a terribly negligent way MCAS. Had they accepted it's a different one, and had a different type rating, all would have been fine.

The 737 airframe reached a point where it was not compatible anymore with modern engines. When the 737 was designed, engines where a lot smaller, the nwer turbofans simply don't fit under the wings of a 737 anymore, so they had to be moved forward. That changed flight characteristics, Boeing used MCAS to compensate for that. If I remember correctly, on-board systems of 737 had issues with handling a second sensor for MCAS (someone with more knowledge please skim in). So they went with that config, they went, as we see in the messages from the chief tech pilot, to forgo major re-certification and thus decided to hide MCAS true nature and impact from the FAA. Consequently, they also hid it from EASA since FAA and EASA basically trusted each others certifications.

Boeing, if you ask me, committed a cardinal sin in aerospace. They cut corners, ignored redundancy, lied to regulators and as a result directly caused airframe losses killing crew and passengers. And that after decades of efforts to increase safety. All that just to save money and maybe keep market share.

Had they just done all the proper testing and development they did after the aircraft losses upfront none of that would have happened.

> Had they just done all the proper testing and development they did after the aircraft losses upfront none of that would have happened

This is a textbook example of how the most expensive point to make a change is after the product ships.

An extreme example, perhaps, but a good one.

And also, the MAX would not exist at all, because the flagship customer (Southwest) had basically said, "If it requires pilot retraining, we won't buy it."

There's plenty of blame to go around, up to and including the society that says it's ok to not pay a living wage to the working class because "infinite downward price pressure is good for consumers" (until they die in a plane crash, that is).

To paraphrase J. K. Galbraith, at any given time there exists an inventory of undiscovered regulatory fraud in the economy, and this inventory is part of the bezzle.[a] In good times regulators are relaxed and trusting, and their approval is easier to obtain. Under these circumstances the rate of regulatory fraud grows, the rate of discovery falls off, and the regulatory bezzle increases rapidly. In bad times all this is reversed. Actions are watched with a narrow, suspicious eye. Regulators assume everyone is dishonest until proven otherwise. Regulatory audits are penetrating and meticulous. Commercial morality is enormously improved. The regulatory bezzle shrinks.

[a] The term "bezzle" was proposed by J. K. Galbraith in The Great Crash of 1929: "To the economist embezzlement is the most interesting of crimes. Alone among the various forms of larceny it has a time parameter. Weeks, months or years may elapse between the commission of the crime and its discovery. (This is a period, incidentally, when the embezzler has his gain and the man who has been embezzled, oddly enough, feels no loss. There is a net increase in psychic wealth.) At any given time there exists an inventory of undiscovered embezzlement in – or more precisely not in – the country’s business and banks. This inventory – it should perhaps be called the bezzle – amounts at any moment to many millions of dollars. It also varies in size with the business cycle. In good times people are relaxed, trusting, and money is plentiful. But even though money is plentiful, there are always many people who need more. Under these circumstances the rate of embezzlement grows, the rate of discovery falls off, and the bezzle increases rapidly. In depression all this is reversed. Money is watched with a narrow, suspicious eye. The man who handles it is assumed to be dishonest until he proves himself otherwise. Audits are penetrating and meticulous. Commercial morality is enormously improved. The bezzle shrinks." (https://www.goodreads.com/work/quotes/1466583-the-great-cras...)

At some point, auditors have to trust some things that their subjects are saying. If the auditor has to re-evaluate every piece of information, they will end up re-doing the subjects' jobs.

There's a bit of backstory here where things the auditors used to verify first hand devolved into from-a-distance. And too friendly a relationship with the businesses they were auditing.

And that's how we got the VAG emissions scandal. Instead of trusting computer outputs, we could have done tried and true dyno tests and the entire thing would have been avoided.

The defeat devices were designed to detect dyno conditions and reduce emissions, it wasn't a case of a regulator trusting a computer output. They were uncovered by doing road tests.

Which a competent regulator would surely be doing randomly across the industry no?

If all they are doing is trusting what the regulated folks are saying they are doing, that isn’t what I would call effective regulation.

Most regulators don’t verify much, (the SEC almost never actually looks at bank account balances,) they just look for inconsistencies in the information they receive.

The SEC isn’t regulating banks generally so bank account balances shouldn’t matter much - they are regulating security markets no?

So looking at discrepancies between data from market participants IS verifying and regulating.

Same as if the FDA looked at raw study data and compared it to equivalent studies for similar types of drugs/treatments, or the FAA had an engineer on staff to double check elements of a new design from a major manufacturer for plausibility

I meant to imply that the SEC could check companies' bank accounts to verify the truthfulness of balance sheets and income statements.

True, though with the way GAAP and accrual accounting works (generally cash balances are only tangential to income, expenses, or even assets once you get past small to medium businesses), and the scale and number of accounts most businesses have, that’s a lot of work for a very muddied view of what is likely happening outside of some specific circumstances like a depository, clearing house, or the like.

That I somewhat like counting the number of bolts ordered and delivered by Boeing to ensure they are staying compliant with building their airframes.

If they never order any, or there aren’t many arriving where you expect, is that a problem? Sure, though it’s probably a subcontractor ordering them, or they haven’t bought any when you were looking because they have a large stockpile, or whatever.

At the end of the day this is what you want, though: people need to realize that they are going to be held personally responsible if they're involved in pencilwhipping the FAA approval process, and tell their boss they're not going to jail so that Boeing can make another 1% profit this year.

I'm not saying it's just the test pilot who lied, or held sole responsibility, but yeah, he was a member of a criminal enterprise that resulted in people's deaths.

I like the result. Too many people hide behind “I was just doing my job and helping the company.” At some point you have to make a personal stand to not harm humanity. And programmers reading this, that era is definitely upon us.

When we see programmers charged who implemented the twisted evil shit Facebook or whoever ordered them to, we will be making some progress on fixing the problem.

and in particular, it appears this guy was actively talking about how to deceive the FAA and avoid attention, so yeah... fuck him, this isn't a random fall guy, this is a chief test pilot who was in on the plan.

again, by all means, this should not be the only prosecution here, and if they can flip him on the rest of his conspirators then he should get a reduced sentence, that's the standard RICO playbook. But prosecution is the lever you use to flip the ones you catch on the rest of their conspirators, and this guy very clearly was a willful participant in this criminal enterprise.

(obviously the jury gets to decide that, but this isn't a courtroom, and barring some gross miscarriage of justice, the quotes they're presenting sound pretty damning)

It isn't one or the other! This guy could have committed fraud and been effectively abetted by a regulatory body that lacks the resources, incentives, or power to actually regulate.

Well the FAA discovered the pilot lied and now the pilot is in serious ** with his career and reputation destroyed: I'm assuming he is looking at possible jail time (not a lawyer).

I'd say this is a serious deterrent to pilots contemplating similar action in the future but I don't think the problem was with the FAA or this pilot. The real problem was the senior management at Boeing who made the conscious decision to put profits ahead of safety. Thus they were directly responsible for creating a culture of short cuts and cheating which lead to the ending of several hundred lives.

This is the reality of most certification processes. There is no government agency that has the knowledge to evaluate planes on a technical level. So what they do is ensuring engineering care and diligence, clear responsibilities and paper trails in a way that risk is minimized and failures can quickly be located and corrected.

At least that is how it is done for medical appliances, I assume FDA and FAA work similarly. But government just doesn't have the extra engineers to technically evaluate every part of a new plane. That would induce massive costs and the manpower simply doesn't exist.

But if companies don't use due diligence to ensure safety, these agencies have the power to penalize you heavily, so you have to comply anyway. Sadly there is also a political component so agencies sometimes have to work against pressure from politicians that don't want to damage domestic brands.

I believe this case was a clear management error for that matter but the FDA probably has more info.

There is such thing as too much regulation. If they had more funding, that would lower industrial output simply by sucking engineers away from productive activity. That’s even if it doesn’t negatively affect aviation with too much regulation.

The principle here should be the same as the principle of proper auditing.

Take the population of x , then sample y and if the sample passes the tests then you have z degree of confidence of projecting the results of the sample over the entire population.

Its not hard. Its only that bureaucrats are generally not hired for the industry knowledge but instead career paper pushers and ticking boxes on their way up, instead of, you know, getting dirty with actual work

Given that the engines were moved forward changing the Centre of Mass I would expect an Aerospace Engineer to be aware and concerned that this is an issue that would require an automated intervention to correct for that, and that would need to be extremely robust.

In hindsight that’s easy for me to say, and the FAA had gone to relying on Boeing engineers, but as mentioned if there was huge pressure for Boeing to compete things could get overlooked.

> The whole point of independent regulation is to not allow obvious conflicts of interest to harm the public.

Like all centralized systems, easy to hack over time. A better system would be decentralized regulatory bodies that check on each other's conclusions instead of a monolithic one.

At this stage you absolutely should not trust any regulatory body, FDA included. (The FDA never replicates any trial for example)

At some level, you have to trust the information you get.

There is also good reason to trust it, especially when it comes from a large company such as Boeing: it is stupid for such an organization to lie to you, because it risks its existence for the rather small payoff of avoiding delays for a single model.

It’s even worse for individual employees at the company: they risk jail time and aren’t even the direct beneficiary.

Example: see above

Government regulation makes sense when there are externalities, for example a polluting factory that harms society but not the company. Airplane manufacturers, however, can self-regulate. If their planes are unsafe they will be bankrupted by the lawsuits of families of dead passengers, and airlines will not buy their planes.

> ...a warning to everyone that that method of regulation is not acceptable.

What part of the situation here is unacceptable? There were, I believe, 2 crashes. We accept more than that with most modes of transport. It isn't obvious that tightening the regulatory process is a net win.

Supposedly, having a serious penalty for lying to the FAA should imply veracity. Same way perjury penalties should ensure truthful court testimony.

Agree the FAA should cross-check, before a plane crashes, but when they catch this behavior they should, IMHO, punish it.

Dance like nobody's watching.

Email like it will one day be read aloud at a deposition.

Reply on news.ycombinator.com