Boeing's 737 Max software outsourced to lower-paid engineers
bloomberg.com
bloomberg.com
The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”
Boeing’s cultivation of Indian companies appeared to pay other dividends. In recent years, it has won several orders for Indian military and commercial aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd. That order included 100 737-Max 8 jets and represented Boeing’s largest order ever from an Indian airline, a coup in a country dominated by Airbus."
I talked to couple of people at the Indian center and they tell me because the center was created due a compulsion and not strategy the quality of work is terrible. Most of them are paid well but have no work to do.
It is explicitly mentioned in the article that the Indian companies DID NOT work on the faulty MCAS system - Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.
This looks like a misleading article trying to shift blame away from Boeing management. Yes, there are problems with outsourcing, which the article brings up trying to show as reason for the problems with 737 Max.
Why is that so hard to understand? Massive fail at risk management maybe?
And they were probably regularly reassured by various internal and external entities that the code was high-quality and thoroughly tested and very safe. Obviously they're not going to outsource if they think it's likely to cause a safety hazard; they just didn't place the odds of this outcome anywhere near what the actual odds ended up being.
The end product may be, but not the design problem process nor the manufacturing process.
The world is changing fast enough that old style management won't get investment after some time.
Managerial bonus or promotion for saving money this quarter >> money lost more quarters out than I'll be around
I have to wonder if seeing SV startups go to multi-billion dollar companies on a regular basis, has given CEOs in other industries a case of envy. But more likely, it's just that pathological optimism works, really well, right up until that moment when it doesn't. Taking risks in order to lower costs makes the bottom line better, which seems to call for more of the same, right up until the moment when the risk explodes in your face.
It is not unlike the attitude towards software security, in many industries. People harping about security risks seems like just all cost and no benefit, right up until the point where you have a big problem.
If so, they're learning the wrong lessons. SV unicorns become valuable because of explosive growth, not because they underpay their engineers. Just the opposite; a lot of engineers at SV startups are paid quite handsomely (especially the early ones at successful unicorns). SV startups don't win on cost-cutting.
The main issue was that there was only one AOA sensor connected in standard packages and the hardware for it failed.
The pitot tubes failed for the doomed Air France incident which caused the software to read incorrectly and stall.
For what it's worth AirBus has a huge engineering department in India: https://www.airbus.com/careers/search-and-apply/opportunitie...
Articles like this are just drummed up to stir negative emotions about "others" (with "struggling to understand English" and "unable to code fizzbuzz") as is evidenced in the rest of the comments below.
Ask Boeing management your question wrt to the hardware failure. Not pile up on software devs
It seems like you’re saying that outsourced devs should have as much accountability as home-based devs or the entity Boeing itself. I don’t agree with this. The salary does not reflect that. The work that’s assigned does not reflect that. Anecdotally working with outsourced devs and based from the comments here, no one honestly thinks that.
It seems like you were triggered by the “excuses” part of the post. But what about the rest of it? He made a valid point (which I think you proved) about people using the article to discriminate.
I guess I’m just wondering why you’re making generalizations. I’ve worked with many devs (who weren’t outsourced) who blame everyone else but themselves. I learned to use ‘git blame’ very early on.
You’re right in that the primary focus of the article was not the outsourced devs being Indian. The issue comes from how people respond to such articles, no matter the article’s intention.
I’m not saying that outsourced devs shouldn’t be held accountable for any faults. The point is that there are different degrees of accountability for all parties involved.
The article only chose to focus on one of these groups, which is fine. What’s not fine is HN comments that seem to not realize this.
No matter the salary or experience or contract, outsourced devs/companies shouldn’t be in a position of power to break critical systems on an airplane. (I’m not saying this is what happened with the Max.)
The negativity seems more targeted at the leaders who arranged this than at the developers. Both the examples you cite are from the same comment, and even that comment is more about resentment towards management corruption than about the incompetence of the developers hired as a result of the corruption.
> The software did exactly as it's supposed to do
I gather new issues with the software have come to light since the accidents, as a result of the heightened scrutiny, and they seem likely to keep the planes grounded for longer than Boeing hoped.
In this case, the lack of redundancy should have been a red-flagged.
This is why domain knowledge is at least as important as coding skills.
I think southwest airlines had multiple sensors
A second AOA sensor is an option.
Which means code was written to account for both configurations.
Many, many things must happen and change to avoid allowing management to make decisions unchecked.
Because this is a case of bargain-bin engineering and tunnel-visioned management ruining a once-reputable company.
Firstly, no the software did not do exactly as it's supposed to. It crashed the plane by rendering it unflyable. It's not supposed to do that. And your description of the MCAS specification is vastly over-simplified - it is not an autopilot and its job is to gradually change the behaviour of the plane in different flight modes (which it must detect), not to directly fly the plane as you suggest (and even two AOA sensors would be insufficient for the type of functionality you describe). The behaviour of the MCAS is absolutely in question, in particular the way it kept "re-adding" the correction, with cumulative effect, ending up fully pitched down.
Secondly, if by "the doomed Air France incident" you mean AF447, your description of it is flat-out wrong. The pilot, not software, stalled and crashed the plane by manually holding the stick back for 15 minutes. The pitot tubes briefly iced over (they didn't "fail") which is not an uncommon occurrence, and the only consequence of this was that the software guardrails that would have prevented the pilot from crashing the plane were removed. All instrumentation systems were fully operational at the time of the crash.
Boeing management did not add a second sensor to some planes and also pilots were not aware of how to switch off mcas.
If you worked in large scale project you would know that most devs are assigned components to work on and usually have no idea what other components are or how they will interact.
This belongs in the true-but-trite category. The decision to depend on a single, unverified AofA input was not a programming error, and neither was the increase in power in the second version. These decisions were made by Boeing, and endorsed by the FAA (to the extent that Boeing informed the FAA of them.)
It would be disturbing if the coders on aircraft control systems were modifying the specifications on the fly, and on the basis of misconceptions like that displayed here.
Responsibility for the dangerous capabilities of MCAS lies with Boeing, which chose to fix an aerodynamic problem this way, and the FAA, which endorsed it.
But in the long term the MBA manager has been likely been promoted and is working elsewhere, killing another company, while his or her CV boasts of "reducing inefficiencies".
Olkiluoto (Finland)
> In May 2006, construction delays of about one year were announced, following quality control problems across the construction. In part the delays were due to the lack of oversight of subcontractors inexperienced in nuclear construction.
> At the end of June 2007, it was reported that Säteilyturvakeskus (STUK), the Finnish Radiation and Nuclear Safety Authority, had found a number of safety-related design and manufacturing 'deficiencies'.
> Initial cost estimates were about €3.7 billion, but the project has since seen several severe cost increments and delays, with latest cost estimates (from 2012) of more than €8 billion.
Flamanville (France)
> In April 2008, the French nuclear safety agency (Autorité de sûreté nucléaire, ASN) reported that a quarter of the welds inspected in the secondary containment steel liner are not in accordance with norms, and that cracks have been found in the concrete base.
> In April 2015, Areva informed the French nuclear regulator ASN that anomalies had been detected in the reactor vessel steel, causing "lower than expected mechanical toughness values". Further tests are underway.[64] In July 2015 The Daily Telegraph reported that Areva had been aware of this problem since 2006.[65] In June 2015, multiple faults in cooling system safety valves were discovered by ASN.
Whatever manager that was, should never be working in an aerospace context, where one design failure or fault can literally kill hundreds of people in minutes (and in this case, did exactly that).
Clearly, Boeing has become infested with people who literally do not understand the fundamental life-and-death consequences with which they are dealing, and who treat everything as fungible commodities. The entire story is essentially about how the outsourcing and commoditizing just expanded and expanded until now, when there were fatal consequences for the passengers, and perhaps even for the company (TBD).
Unless this changes, this is no longer a company who can be trusted with such life-and-death engineering decisions.
If the sentence was “the mean salary of $9/hr” then I would be thinking differently.
Is your issue just with the amount ($9) being paid?
To be clear, we paid $120-150 per developer and L&TI paid them nothing. You’d see .NET code with a single class that has a few thousand lines of code, no concept of unit tests, integration tests, manual or automated acceptance tests, usability tests, etc.
Shelf life of most of this garbage was a few months if something ever did get delivered. This went on for years.
The whole industry is corrupt.
Of course generally the reason people go along with the BS is that either they're incompetent or mediocre and BS suits them just fine, or they've given up and are just happy to be swept along and collect a paycheck.
>You’d see .NET code with a single class that has a few thousand lines of cod
If we are blaming entire nationalities, I have seen code written by an American. His idea of refactoring was copy paste the function, append a '2' to the function name and use it. There were at least a 50-100 functions like this in the project that had like 30-50 source files.
I have one person in our Mumbai office out of a 8 person team I manage. He had been absolutely fantastic with the work he does.
As an extreme example, one ex-employee stopped working one day because his wireless keyboard did not have batteries (a kB connected to his work MacBook Pro). Colleagues told him to just buy some AA batteries while I was back (I would have refunded them to him) and he did not want to do it.
Nowadays if I see HCL or Tata in resumes here in GDL, it is an automatic pass.
LOL! Ahem. You'll discover that they get someone competent in for the phone interview and you'll later discover that the story is now that the person you spoke to was representative - in terms of skills, experience and competence - of the sort of person who'll actually doing the work, and is not the one doing it. Later still you'll discover that this is not the case.
Still, they're very cheap, aren't they?
In all my years consulting for the companies I have never been "client interviewed"
The software got the reading that the nose was too pitched up and to prevent stall it pitched the nose down.
The software could handle reading from multiple sensor which Boeing sold separately.
It's an American management decision to make more money, not a $9 an hour software engineer's job to decide how to fly the plane.
Coming up after the break: How EA loot boxes are programmed by $10 an hour cheap software engineers from India. Grab your pitchforks on aisle #3
As you say, you're commenting without having any experience with the issue, but, as you can imagine, there's very little overlap between cheap and good.
In any case, even if we accept your premise, my point was that assigning blame to the person you hired rather than your process of hiring that let a large team of unqualified people work on your critical software is odd. (Not saying you personally are blaming them, but that was my general point)
>but, as you can imagine, there's very little overlap between cheap and good.
Why? Even outside of software, for e.g. my wife is a post doctoral scientist and she gets paid a pittance for working 9 sometimes 10 hour days doing critical research. Somebodys wage being associated with the quality of their output only happens in a small cross-section of industries. I work in pharma and while I'm happy with my compensation, its a fraction of SV startup salaries. I know.. this might come across as super salty, but I don't think their wage is in anyway co-related with the quality of their output.
/s
The main problem was higher up. The overall design was bad, and Boeing didn't seem to have a handle on it.
Any competent engineer would raise moral and ethical questions when working on software that is "mission ciritical". Causing a passenger plane to continually go into a nose dive because of data coming from a single sensor is absolutely ludicrous. Just because the software was horribly designed, doesn't mean it isn't a software issue.
1. https://www.seattletimes.com/business/boeing-aerospace/a-lac...
I used to watch PR videos from Boeing on new planes being tested and feeling sense of wonder and awe. Now I feel like puking. Of course not the rank and file. But the ones in corner offices.
The inescapable problem is that all newer planes from Boeing and likely other manufacturers are suspect in uncountably tens of thousands of ways because they cannot be presently assured (without engineering reviews, inspections and documentation) to be fit for passenger travel due to the lackadaisical regulatory climate in which they were developed and manufactured. The 787, 737 NG (-600 and up) and 737 MAX are just the tips of the known problems. And now Boeing wants a folding wing plane too?
It's not engineer that or this. It is MBA this and MBA that.
>Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.
>Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.
Even inside US, Indian engineers are not allowed in critical software that goes into defense and (most likely) airlines, only citizens can do that. So, there is no way they would have outsourced safety critical stuff to HCL.
As for code quality, this team was working on a Spring / Java application. They were instructed to use Spring JDBC templates instead of Hibernate. So of course they build their own query-builder that was just concatenating strings. Naturally we paid them to clean up their own mess.
Heh, basically funding external developers too learn without the benefit of retaining them then.
That need not be a disadvantage, I've seen awesome self-taught coders (and really bad ones too).
This should be a company-ending event for Boeing. These folks should not be making aircraft used by living, breathing human beings.
The stock holders Boeing is beholden to, that clamor for this cost cutting, should pay.
This isn't cheating on emissions. We have to send a strong message.
Losing 1000 people is a tragedy, and those responsible should be held responsible. But a company like Boeing is a huge machine with many people working diligently on extremely complicated products, the vast majority of whom didn’t touch this faulty system at all. Which is to say you shouldn’t throw the baby out with the bath water.
Boeing is dead to me.
With Boeing trashing its reputation and Airbus sitting on an enormous backlog I can envisage Chinese airliners making inroads in Africa and Asia. But to do so they need to make solid, dependable products; they can't scrimp or cut corners in pursuit of short-term profit.
I guarantee you a lot of the legally binding design and testing documents were signed by engineers, maybe some product manager types. A few QA folks? But that's the extent of it as far as people who have the clear cut legal liability. You want to get anyone else? You're likely looking at a massive investigation, and spending years or even decades looking for incriminating documents or emails. And unless I miss my guess, the government is just not terribly likely to find any.
And give them to the family of the victims, ON TOP of whatever punitive damage Boeing will be paying.
Holy crappus, what was Boeing thinking?
In my mental model of this kind of development (which could be wrong), you'd absolutely not want the 10x hotshot coder, you'd want predictable worker bees to faithfully transcribe from one specification format to another specification format slightly closer to the metal (or to a verification tool) until you eventually have a product that is working as designed. And from what I have read so far the MAX software does work as designed, the problem already started with garbage in. The article reads like an attempt at shifting blame that I think is entirely uncalled for.
Which gets you exactly what happened here. The 10x "hotshot coder" isn't a coder who vomits out 10x more code than another, but the coder that knows the right code for the job and also understands the provenance of from whence the data operated on comes from, and to whence it goes.
Software engineering and computer science is about so much more than coding. It's about knowing the types of problems an implementation can solve, what boundary conditions have to be taken into account, and knowing what code isn't the right code. All for any subject matter area they end up operating in.
You'll be amazed the speed ups you get from not having to write ten different versions of code implementing the same functionality before finding the one that plays well and reads well with the rest of the system. All that extra time spent not coding can be spent going over the operating environment to make sure the specification you were handed is actually complete.
Unfortunately, I have the feeling Boeing would have hidden that information regardless seeing as their design doesn't even necessarily comply with 25.173 as written without interpreting it as allowing a computer to magically make an unairworthy frame airworthy. But that is beside the point.
Once your management starts assuming the process will spit out product as high quality without your expertise, you can pretty much set up a timer to measure the time until the first impedance mismatch from a false assumption not rooted out rears its ugly head.
What isn't well understood is that software is never fully spec-ed out. No matter how hard some businesses may try, only the coder is in the position to find corner cases, unspecified cases, cases that "shouldn't happen" (but can), and just bugs in the spec. It's unrealistic to build a quality system without a feedback loop between the people writing the spec and the people coding the spec. Both sides have to be smart and engaged.
There's certainly nothing wrong with the article. I didn't see it taking a point-of-view on who is and isn't to blame. It's telling factual information, which is relevant to the story of what happened, and you can make your own judgments.
On a side note, if anyone is looking for all-around engineers, including software/electrical/mechanical stuff, feel free to shoot me a message.