Comment by V_Terranova_Jr | Hacker News Reader

Comment by V_Terranova_Jr | Hacker News Reader

Parent Full thread

Having worked with Boeing on other aerospace programs (not commercial though) and having been on the Gov side, the basic findings ring true. Boeing's culture really is as flawed as the report reads. The idea that Dennis Muilenburg, however, originated or did more than prior company management to foster this culture, is nonsense. They remain, even today, operating with this culture. The real problem that stands in the way of "fundamental structural reform" is that regardless of specific aerospace market, few alternatives to Boeing exist. And don't think the corporate cultures at other traditional US aerospace primes is consistently better.

The points about Government acquiescence to Boeing pressure in performing regulation also resonate. Beware the tendency to make this a single-axis "more vs. less" regulation issue. The solution isn't "more regulation". The central concern should about regulatory culture. Ultimately, responsibility lines must be drawn, standards established, adjudication performed, and unique or specific situations accommodated. Inevitably, the "less regulation" crowd corrodes the kind of regulatory culture that serves the best interests of the populace in these processes. Good regulation depends on having highly-competent, wise, empowered, and apolitical persons on the Government side. While providing regulatory organizations lots of funding doesn't ensure that they hire, empower, and maintain such a cadre, starving them of resources and implying the Government can never be competent, so it should reflect the role back to Industry, is pathological and will produce outcomes like the 737 Max.

We need to not just empower and provide adequate resources to regulators, but also demand and foster a culture of competence and good judgment^. Making that happen is a lot harder than arguing about "more or less", but necessary for proper outcomes.

^"good judgment" here explicitly excludes decision-making with regard to political implications

Can confirm from first-hand experience: I worked for NASA for about 15 years (1988-2003) so I got to see how the aerospace sausage is made. Everything you say rings very true to me. It's actually kind of amazing to me that there aren't more problems like the 737-max. Even in my day there were strong dis-incentives for anyone to raise a red flag when they saw corners being cut or decisions being made for political rather than technical reasons (which happened a lot). That's ultimately why I quit: I was faced with a very stark choice between doing what I thought was the Right Thing and torpedoing my career, or keeping my very cushy and vert well-paid job and becoming part of the problem. Apply that incentive structure to a few hundred thousand employees and the result is MCAS.

Can you elaborate on these dis-incentives? Are they systemic or individual?

I work in veterinary medicine production and I am a little concerned by some of the features of quality control in my workplace. Basically, if a mistake is made, there is an additional load of paperwork and investigation that has to occur according to procedure. If you never admit that a mistake was made, you can save yourself a lot of grief and leave work at 5pm instead of staying later. I think the culture is actually fairly positive, a lot of mistakes do get reported, quality assurance does try to keep the paperwork manageable. But there is massive resistance to change in processes for even minor things. Some people do have a bad habit of avoiding the proper channels when they think they can. I suspect this is normal friction for any workplace that has regulatory compliance.

> Can you elaborate on these dis-incentives?

Hoo boy. How much time have you got?

The fundamental problem with NASA is that one of its main functions is to serve as a pork funnel to send money to key congressional districts. This goal is, as one might imagine, sometimes at odds with NASA's other mission of, you know, exploring space. When these are in conflict, the former wins. In the private sector there is a similar dynamic, but instead of the pork funnel tail wagging the engineering dog, it's maximizing profits that is the primary driver.

So what happens at some point in everyone's career is that they are given marching orders that serve the financial goal but not the technical ones and they have to decide what to do. If they raise objections they are labelled as "not a team player" and they don't get promoted. So the ranks of upper management are filled with people who are willing to play the game while the troublemakers are shunted off onto career sidings and turn into grumpy old men with no power or influence.

In my case I found myself in a very bizarre situation for two reasons. First, I had credentials that were visible outside the organization. I was a researcher with a pretty good publication and citation record (in fact, according to CiteSeer, I was the most cited CS researcher in all of NASA, and I held that title for about ten years after I left). That made it hard for them to fire me. And second, I had a patron, a senior manager who liked me and got me promoted over some pretty strenuous objections. So I ended up on the highest rung of the technical career ladder, essentially untouchable. But I had also made some powerful enemies. So what happened to me was that my funding dried up.

A small digression: the process of funding research is largely a sham, though I didn't realize it at the time. Ostensibly, people write proposals which undergo peer review on their technical merits (though of course what really happens is that the money is assigned based on which congressional district your institutional affiliation is in). One of the weird things about NASA research proposals is that you have to tell them what results you are going to achieve, which is kinda weird because if you know that ahead of time it kinda stops being research. Also, as part of this year's proposal you have to tell them whether you accomplished the goals from last year's proposal, and that goes into your score.

I came up with what I thought was a clever way to game the system: I wrote proposals for work that I had already done, which insured that I would always be able to provide accurate predictions of the results. It turns out that everyone does this, but I didn't realize it at the time. So one year my proposal was rejected on the grounds that what I was proposing was too ambitious, that it could not be done on the budget I was requesting. Which, of course, I knew to be false because I had already done the work the previous year, but of course I couldn't say that. It was only many years later that I came to realize that the reviewer already knew it too, but that this was a not-so-subtle signal that I needed to toe the party line.

But I didn't, so I found myself being assigned to projects that had some extra money in their budgets to cover my salary but where I was essentially a potted plant. (I spent an entire year shepherding purchase orders for search appliances through the procurement system.) Towards the end I was spending my days in an enormous windowless office by myself with literally nothing to do. That's when I quit.

Wow, thankyou for an excellent answer to my question :)

I have spent some time in biological science research and your comments about grants holds very true. The result is that a lot of money is funneled into inefficient laboratories producing safe and often not very useful research, that is when they aren't hording results for a Nature paper that will never come. Sometimes this benefits some truly terrible people who are ruthless in their goal of climbing up the ivory tower. There are still plenty of great people in academia who are doing well, but I wish medical research at least tried to be as meritocratic as it claims to be. Right now the grant committees are an old boys club.

> Wow, thankyou for an excellent answer to my question :)

You bet.

> biological science research

Yeah, similar dynamics play themselves out in all human endeavors. The Ford Pinto gas tank. The Corona Virus. Sub-prime mortgages. It's all examples of the bean-counters and the good-old-boy network overruling the technocrats. Welcome to the human condition.

I noticed from your bio that you worked at JPL.

Is any of this unique to JPL (or at least more extreme there) due its organizational structure that is somewhat different than other NASA Centers? I’ve sometimes heard of JPL being “quasi-NASA” and thus playing by a different set of rules

If anything, I think JPL is probably better than the rest of the aerospace industry in this regard. I think JPL actively tries to shield its technical people from some of the ugly politics playing out behind the scenes. In retrospect, I was definitely shielded from a lot of it for much of my career, particularly early on. That's a big reason my career was able to advance despite being so naive for so long. I think if I had been at Boeing I would have gone down in screaming flames much sooner.

But you also have to keep in mind that this was all a very long time ago. I'm offering my testimony here not so much to inform current events as to vouch for the plausibility of what V_Terranova_Jr said in the root comment of this thread.

I worked for the "Canadian JPL" back in the previous "climate change" panic, "the ozone hole."

Even though it was a government agency, working at a national lab felt even more productive than the private sector.

The reasons are that the departmental leadership is 100% done by research scientists who are ruthlessly efficient at doing science and writing papers about them.

My department had approx. 4 senior scientists, a computer hardware engineer, a payload engineer, a CNC technician, a sysadmin, 2 technical consultants, and one administrative staff.

We did custom payloads for the Shuttle, balloons and rockets which did atmospheric measurements. The software was primarily custom in C for telemetry and HP Basic/etc. for plotting.

I highly recommend working for a national lab for a couple of years. Besides an unlimited budget, it's great having a sense of mission.

Factoid 1: We used Fidonet to copy data from launch sites to the lab for quick turnaround and to ensure the instruments were functioning and calibrated properly.

Factoid 2: Since this we pre-WWW, sample source code was often hard to obtain. So we had the world's shareware on a single optical disk with an index file so people could find sample code.

Factoid 3: But since it was a national lab, if we needed, we had access to SOTA civil and military systems that came with source code. I was familiar with those systems, but we never needed them while I was involved.

https://en.wikipedia.org/wiki/Ozone_depletion

What was more tractable about studying the ozone hole than climate change was that we could make direct observations and reducing CFCs had a fairly immediate effect. There wasn't an issue with "ozone hole deniers."

To see the lab's papers, do a scientific search for Dr. Hans Fast and his co-authors.

Thank you so much for this. I’ve been looking into applying to a few national labs and what you’ve started about the “mission first” culture is exactly what I’m looking for

I was going to say, wouldn’t you do what Feynman did... but that’s what you did:

“I wrote proposals for work that I had already done, which insured that I would always be able to provide accurate predictions of the results.”

Yep. I was told very recently by someone in a position to know that "everyone does that, didn't you know?" No, I didn't.

But what's the point then? I dispair. I can see the logic in paying people after they've produced outcomes. But why not just say that?

There are a lot of truths that go unsaid in human society. Consensual self-deception seems to be part of the human condition. It's not really cause for despair, just part of how the world is. There are a lot of other things about life that are sub-optimal, why let this one get you down more than all the others?

That kinda explains why NASA projects are always outright petrified at the idea of accepting -any- risk at all.

To be fair, a lot of NASA's risk aversion stems from the fact that their work is very capital intensive. Even a cheap spacecraft costs many hundreds of millions of dollars, so failures can be very, very expensive.

Also, in the case of the manned program, human lives are at risk.

NASA risk aversion seems directly related to their patrons' priorities.

What's the cost of NASA not going to the moon yet? Zero. What's the cost of NASA failing at going to the moon? Large.

Consequently: the organization optimizes to avoid failure at all costs, even at the cost of minimizing success.

On the other hand, what was the cost of 1960s NASA of not going to the moon? Large. The cost of failure? Large, but slightly less.

Result: we went to the moon.

Great write up. Can you say more about how the congressional district pork funnel works? I mean, how brazen is it? What are some examples of the kind of projects that serve little technical purpose and exist only to funnel money to a district?

> Great write up.

Thanks.

> Can you say more about how the congressional district pork funnel works? I mean, how brazen is it?

That depends on what you mean by "brazen". In my tenure no one ever explicitly said, "We need to do X because we need to send money to congressional district Y." It was more like: "Your proposal will stand a much better chance if it's a joint proposal with institution Z." When I asked why, there was never any technical rationale for it. It was more like, "Look, I'm just trying to help you out here. Don't ask too many questions."

To be fair, sometimes those coerced collaborations were productive. But other times they caused some very serious problems in the case where there was really nothing useful coming out of Z. In that case we had to find a way to shoehorn something coming out of Z into the project, and that usually did not end well.

> how brazen is it?

Current fighter projects try to hit all 50 groups of senators, er, states.

> "good judgment"

I'll add the concept of 'good faith'. I feel like fashionable neoliberal economics fails because you can't incentivize good faith. And indeed the amoral foundations is corrosive to it. And here we are.

> you can't incentivize good faith.

Thank you. I have been looking for a pithy way to articulate this idea. There is literally no metric or combination of metrics that can perfectly incentivize a person who is... morally challenged.

> and foster a culture of competence and good judgment

I agree.. but I don't think you can do that organically. I think part of the issue is the size of the corporations, the concentration of that corporate power, and the inability of the market or regulatory agencies to function correctly when this is the case.

You get cultures of competence and good judgement when there is a real competitive market with multiple strong players each vying to have the best products.

I should have been more clear and said "foster a culture of regulatory competence and good judgment".

This is certainly achievable with the right leadership, wide-basis political protection (cannot be achieved when an agency's leadership is subject to unprincipled undercutting by the President), and sufficient resources.

>You get cultures of competence and good judgement when there is a real competitive market

This doesn't even happen in markets that are highly competitive. Only professional and strict engineering standards work.

Not voodoo economics.

> Beware the tendency to make this a single-axis "more vs. less" regulation issue.

Nobody does this, not even the most outspoken anarchist libertarian. The question is not more or less regulation, the question is who does the regulation. I would argue that regulation should be done by those who stand the most to lose. This is not corruptible Government officials but by insurance companies, airliners, pilots and passengers, i.e. by those who have skin in the game.

If anything, the Government regulation makes the situation worse, because by its nature has incentive structures that are completely inadequate. Perhaps if damages were payed out of the compensation of employees at the FAA the situation would be different. Perhaps if everybody at the FAA were earning minimum wage after the 737 Max crashes and all that money went to the families of the crash victims, then there would be an impetus to change the culture in the FAA and the aviation industry. Until the regulation is done by organizations that have skin in the game, I would say the FAA is doing more harm than good, by lulling the public into a false sense of security.

The FAA does have skin in the game; you're just discounting it.

The FAA's track record led to a situation where for a good portion of the world, the FAA's word was considered good, and bidirectional approvals between the local and American aviation regulatory agencies were common. This allowed the capability to run a much tighter and cheaper ship industry-wide, because companies could rest assured that their FAA papers were de-facto gold standard everywhere else.

It remains to be seen whether that will remain the case, and the reputation of the agency has been heavily damaged by the MAX disasters, especially given how the FAA were the last regulatory agency to ground the plane, provably through political machinations.

As long as the FAA continues to function as a vestigial appendage of the American Aerospace industry instead of as the last word by the most competent the system can muster, there will be a magnification of the costs of getting new aircraft approved as local regulators crack down.

> Nobody does this, not even the most outspoken anarchist libertarian.

Listen to any debate aired in the press about regulation and it almost invariably degenerates into "more vs. less". This is consistent with U.S. political discourse in general, which turns major issues into single-axis ones. It's the nature of politics these days: for vs. against, more vs. less, left vs. right. Somehow everybody unquestioningly gets locked into this broken, simplistic framing of inherently complex issues.

Sure there are some words thrown in to make the argument sound more complex, like "too much burdensome regulation" but usually the remedy being suggested is to "cut regulation", not asking how we do smarter regulation, what the enforcement situation looks like, and what systems we put in place for periodic re-assessments of the regulations. Nor to suggest that at the end of the day, if we don't have good people in these regulatory organizations, no amount of process can make up for that.

> If anything, the Government regulation makes the situation worse, because by its nature has incentive structures that are completely inadequate. ... I would say the FAA is doing more harm than good, by lulling the public into a false sense of security.

The tremendous track record established through the regulatory efforts of the FAA is starkly opposing evidence against this line of argument. Everyone who flies commercially around the world should, while disappointed in their failings, be happy at the overall magnitude of public benefit the FAA has ensured. Every time I get on a commercial flight I'm amazed at how we operate these incredibly complex machines with the kind of safety and reliability we enjoy. But it's no happy accident or dumb luck.

The thing I don't like about just blindly labeling all government as inefficient bureaucrats is that if you actually spend time on the inside, you realize there truly are a lot of people that care and want to do the right thing. There are some truly sharp, talented, well-motivated people who take their responsibility to the public extraordinarily seriously.

> I would argue that regulation should be done by those who stand the most to lose. This is not corruptible Government officials but by insurance companies, airliners, pilots and passengers, i.e. by those who have skin in the game.

The people that have the most to lose are the citizenry. The Government works on their behalf on most matters of safety. And for the most part in democratic nations with low levels of corruption, this system generally works well. In the current construct, money and a "getting in bed with the regulated is good for the people" mentality among politicians are the corrosive factors. Do you really think that some kind of joint body comprised of pilot and insurance companies is somehow less inherently corruptible by money and its own sphere of politics? I'd argue there would be even less sunlight to expose conflicts of interest.

The radical reform we need in the U.S. is to contend with politicians working for the interests of those other than the citizenry, including the misguided idea that when there is tension between industrial economic interests and the public interest, that they should by default favor or err on the side of industrial economic interests. Channeling Upton Sinclair, this will probably not change until the nation finds ways to further mitigate the effects of money on politicians. The well-meaning mid-level career civil servant cannot work on behalf of the people if he is being undercut by the political appointees at the top.

Reply on news.ycombinator.com