Hacker News Reader

Comment by pcl | Hacker News Reader

> Because the fault was triggered by specific streams of erroneous flight data, a new software patch can be devised that monitors the computer for that highly unusual condition and prevents movement of the stabilizer when it occurs, one of the people said.

Oh man. That sounds like a mess.

View on HN

That sounds like adding a heuristic which works most of the time which is something I can see myself doing to cover some pathfinding issue in a game where it's sufficient if it works 99,9% of the time or when working on a simulation where I can look on the result data afterwards and can see if something looks highly unusual despite the heuristic and improve it iteratively - a luxury you don't get with air planes, at least not without casualties. Additionally given the very limited computing power of the hardware in the 737 adding even more software to it might create even more problems.

Yeah, it’s a nice way of saying “this is a hack”.

Move fast and kill things. At least that seems to be the motto of self-driving carmakers, too.

Wasn’t aware the compute power was low on the 737. Interested if you have some more info?

Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?

The issue is (from what I understand as layman) that to avoid recertification they kept their old computer hardware from earlier 737 generations, and added more and more tasks to do for it.

https://www.nytimes.com/2019/06/26/business/boeing-737-max-f... (somewhere in the middle: "The issue discovered last week is linked to the data-processing speed of a specific flight control computer chip, according to the two people with knowledge of the matter. In the test, the F.A.A. pilot encountered delays in executing a crucial step required to stabilize an aircraft." )

That's surreal - avionics should run on hard real time. (Whether it's offline scheduled and thus "impossible" to miss a deadline, or if it's "soft" realtime and merely testing ensures this, is not the important part. The important part is that it should never miss a deadline.)

I can understand it looks weird but it isn't. A faster processor has no more real time guarantees than a slower processor from the 80s

Of course, the faster one can do more stuff in less time, but for flying an airplane (or going to the moon) the amount of computing power needed is much less than people imagine.

The issue here is adding more computing into an aging platform. But there's nothing inherently wrong into using an older platform into the flight computer.

Now I'm not sure if for certification purposes they would be allowed to replace it with modern hardware but just keeping the functionality the same, I believe the answer would be yes, that they are allowed (but they probably didn't want to rewrite the sw and recertify the hw)

Processing has very little to do with hitting realtime deadlines. Either you do or you don’t. In fact, sometimes faster CPUs are worse for realtime because their latency can be worse.

is it me or the certification looks at the hardware but not at the software ?

Certification mostly looks at the behaviour, how you get there is your IP for the most part

For example, the regulator doesn't care (much) how you built your wing, but it does care that's able to withstand 150% of the maximum expected forces.

You can have parts done by different manufacturers (but to the same spec) and this is sometimes done on purpose in redundancy settings so that unknown bugs in one model are less likely to show up in a different model.

> to avoid recertification they kept their old computer hardware from earlier 737 generations,

This is unbelievably fucked

No, it's not. Certification is an extremely expensive process with dubious benefits for safety. Why would you change hardware that has worked fine if it can fulfill your requirements?

That's related to an issue often found in medical devices development: the certified toolchain contains errors, but you cannot use the newer version with fixed errors because it is not certified. Of course, the new toolchain passes the same certification suite as does the faulty one (because the certification body does only check the documentation for completeness and basic plausibility and does not have any means to test anything themselves), it's just that it isn't commercially viable to "buy the official stamp".

Because it obviously did not work fine and does not fulfill the requirements.

What seems to have dubious benefits for safety is lying to the regulators and shoehorning new processes into old technology. Because hundreds of people have died from that exact thing, just on this one airplane.

The airplanes didn't crash because the computer was too slow. They crashed because safety critical data was read from a sensor without redundancies.

But now during simulations test pilots apparently found situations where the computer was too slow to handle their input in adequate time. So while the crash was not the caused by this problem, there seems to be a problem.

> No, it's not. Certification is an extremely expensive process with dubious benefits for safety. Why would you change hardware that has worked fine if it can fulfill your requirements?

Is it fair to say the hardware is fulfilling the requirements though?

If the requirements were for it to buck like a bronco before slamming into the ground then I say job well done.

Here's an article about it. According to this piece, 737 avionics run on a pair of 80286s.

https://www.moonofalabama.org/2019/06/boeings-software-fix-f...

If true it uses a 286, for some relative comparison, the fastest 286 is 1.5 MIPS. A run of the mill current Arduino is probably 20 MIPS, and a Raspberry PI3 is 2451 MIPS.

Can’t find the exact MIPS estimate for the Harris CS80C286 25Mhz 286, but given that the 1.5MIPS was quoted for the 10Mhz Intel one I’d guess it’s substantially faster.

Although still slower than an Arduino :)

I love how HN is taking umbrage in the rest of the comments about Boeing being too aggressive with MCAS et al. redesign.

Then taking umbrage here that they were too conservative and possibly froze the hardware spec.

The general consensus I've seen here on HN is that Boeing should have done a clean sheet design rather than trying to shoehorn increasingly ill-fitting features into a decades-old design. You're not accurately representing that view.

http://news.aa.com/news/news-details/2011/AMR-Corporation-An...

"American also intends to order 100 of Boeing's expected new evolution of the 737NG, with a new engine that would offer even more significant fuel-efficiency gains over today's models. American is pleased to be the first airline to commit to Boeing's new 737 family offering, which is expected to provide a new level of economic efficiency and operational performance, pending final confirmation of the program by Boeing. This airplane would be powered by CFM International's LEAP-X engine."

When a customer demands...

No umbrage here. Just interesting to me. My Commodore 64 was half a MIP.

The should at least get 1202 error..

NASA supposedly bought 386 chips from eBay

The lesson is real engineers do not need row hammer afflicted processing power, they are not running crappy JavaScript. They built something that works and over engineered the hell out it.

Ford in Detroit once lost a mainframe. For 20 years

Yep. At this point in time they should have plenty of test cases and regression tests which should catch such situations. While Boeing may feel they are doing a lot of testing in the simulator - there are way fewer simulators staffed with test pilots than there are 737 MAX out there. When such errors surface at this time in manual testing it calls into question the development process. And a lot of the trust - aka. certification - is based on that process.

As I have mentioned here before: The angle of attack sensor was never intended for the purpose it is now used for - military aircraft may do such things but not as an afterthought and not connected with a control surface which has a lot of delay (trim wheel winding. Delay is a major factor in destabilizing control loops). Equivalent sensors are the airspeed pitot tubes but they are a much more robust concept. Even they have brought down airplanes. We lack sufficient real world (heat, humidity, snow) information about the quality of the sensor data even before it may get distorted by transmission and processing on its way to the control law. Cleaning and validating sensor data is tricky under the best of circumstances. It does not sound like they have a handle on it.

Hi Boeing, what real world data from mounted angle of attack sensors have you analyzed? Why do you believe this data sufficiently represents climate zones and flight conditions (take-off, landing, turbulence, malfunction modes)? How do you clean and validate the sensor data? Please explain, because otherwise I won't step into one of these machines.

Can you forward the data Airbus, your car or public transport vehicle manufacturer, Otis or equivalent elevator company obviously sent you for their respective operating environments?

Wow a textbook example of a bandaid solution. Reminds me of something a novice programmer would do. When faced with a complex problem....nothing beats an even more complicated solution!

When your design is a mess you're probably better off to scrap it altogether and start from a clean piece of paper. "Fixes" to a bad design are adding to the mess. And in this case the mess has already cost hundreds of lives. You cannot "patch" this...

I see this sentiment a lot, but its kind of surprising. Traditional wisdom states the opposite - never do a full rewrite, you lose the years of buit-up fixes and edge cases and have to learn from scratch in too many places.

Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety.

Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.

The problem here is that what Boeing is selling ("737 that spends significantly less fuel and doesn't need recertification") simply can't be made safe. Whatever results from the marketing driven goal (selling something that doesn't and can't exist in reality) will remain unsafe as long as all the "already sold features" are preserved.

So it's not if they should "start from clean" but just accept the need for the new certification and then do the changes that are really needed. It will be less profitable for Boeing but it righly should be this time! They are killing people otherwise.

The funny thing is this plane already exists. It’s called the 757 and Boeing stopped making it 15 years ago because the airlines wanted stretched 737s far more than they wanted the clean sheet redesign.

It's one of those fixes where they add up the value of human life if something goes wrong and say 'OK we can handle that, ship it"

I wonder if the human life calculation worked well this time. As far as I see, Boeing lost more than the sum of the human lives; they also lost reputation for everything new they’ve designed in the last 7 years being corrupted, and they also engulfed the reputation of FAA with them, whose agents would fit the definition of “corrupted” by any people’s definition (I know, they are not, they just used agents of Boeing to inspect Boeing because they were understaffed), and the FAA showed the last step of failure by not admitting that the plane had to be stopped until a few days after the European agencies.

In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.

There is so much of U.K. which is incredible and depressing at the same time. Completely pummelled by war, (receiving no aid like West Germany did) they managed to, in 1949, release a Space Age passenger commercial airliner, defining the genre. It looks modern even today.

Yet, fumbled the ball.

They also had a space program, closed it down after their first satellite, which was a success.

Concorde! (With France.)

It goes on an on, I'm sure you have favourite examples.

It's heart wrenching.

Generally short-sighted politics gets in the way in the UK based on my own opinion. Support/budgets/etc get withdrawn at precisely the wrong moment (and usually for some petty point-scoring or if ministers are running scared from the media/opposition)

Its happening right now too with electric cars and solar electricity - solar has generally been seen as a success with a lot of installations taking us to 3rd place in Europe for solar megawatts despite there being essentially no sun (1), and EV cars are starting to become something actually viable for most people's lives. The government has just announced that the UK is now legally bound to be carbon neutral by 2050... yet the government chose also to revoke funding and support for both solar and EVs and so subsequently update has plummeted (2, 3) right at this vital point.

1 - https://en.wikipedia.org/wiki/Solar_energy_in_the_European_U... 2 - https://www.theguardian.com/environment/2016/apr/08/solar-in... 3 - https://www.autocar.co.uk/car-news/industry/new-car-sales-fa...

“The largest recipient of Marshall Plan money was the United Kingdom (receiving about 26% of the total), followed by France (18%) and West Germany (11%).” https://en.wikipedia.org/wiki/Marshall_Plan

Since others may be curious, what the UK received under the Marshall Plan looks to be about 3x what they paid back under Lend-Lease. (Which in turn was maybe 10% of the cost of what was shipped.)

You may like this take (rant?) which describes further loans from US to UK. https://www.bbc.co.uk/history/british/modern/marshall_01.sht...

Interesting factoids:

Marshall plan replaced and thankfully rejected the previous Morgenthau plan - which would have explicitly ruled out any aid to Germany, and actually further destroyed the country.

In a fine bit of statesmanship Marshall plan aid was offered to the Soviets, who rejected it. It was offered in fairness, but in the certain knowledge Stalin would refuse. It would never have got through Congress anyway. :)

Today I learned!

Last week I was at Munich, with +40 other Europeans: French, Spanish, Italian, German, Dutch, Swedes and of course Irish and British. As I was sipping my beer at a biergarten I realized we were all speaking English, in Germany, surrounded by people whose native language wasn't English, right when Brexit is about to happen, just when any British could say: we won. But no, as you said, they just fumbled the ball, again.

I thought the US won..

The United Kingdom received over twice the amount that West Germany did under the Marshall plan. https://en.wikipedia.org/wiki/Marshall_Plan#Expenditures

The U.K. had an educational computer initiative with the BBC and Acorn. Worked wonders. Then they discontinued the effort.

Yes, although ARM came out of that and now look at that (...or your smartphone) so its not all a disaster.

The Raspberry Pi is a spiritual successor which I am generally pleased to see. The RPi was I think partly responsible for a real Computer Science curriculum getting back into schools ... I hate to say it but I think it was Michael Gove that made this happen!

Arm Holdings is now owned by Japan's Softbank, because a Conservative (business-first) administration thought selling off the family silver was a fine idea.

This is pretty typical of the UK government's approach to the computer hardware/software industry. Had the first business computer industry in Europe (anyone else remember the LEO line of mainframes?) and a promising software biz, of which only bits remain. (The gaming industry is thriving, but the rest mostly ended up being absorbed by US multinationals. I could speculate that having a common language, in combination with a series of governments who believed in leaving things to the Invisible Hand, and a business culture dominated by accountants, were at the root of the problem ...)

Coemergence of government competence in invention and supporting application seems almost as rare as companies delivering on a second blockbuster idea.

Which kind of speaks to the prevalence of luck over design in favorable outcomes at any macro level. Too many small, randomly whirling bits, and they all need to line up just so.

TSR2, Hawker P.1154 (Supersonic Harrier)

Royal Aircraft Establishment - merged and amalgamated a dozen times to become DERA - privatised. Now QinetiQ.

Most of the decent research sites are gone now.

GEC/Plessey

I'm going to stop now, it'll get depressing.

Dyson were doing quite well till Mr Dyson went pro brexit and moved the company to Sinagpore.

DeHavilland didn’t do that badly - the last Nimrod (based on the Comet) was retired in 2011. The name disappeared because, like a whole host of British aviation manufacturers (English Electric, Folland, Vickers, Supermarine, Hawker-Siddeley, Bristol, Hunting, Avro), they successively merged together until all that was left was a name you probably are familiar with - BAE Systems.

Except that, unlike de Havailland, Boeing is too big to fail. There's just no way the US will allow it to keel over, and leave Airbus the sole supplier. And as with the banks, you can bet that this implicit guarantee gets figured into what decisions people take.

https://en.wikipedia.org/wiki/De_Havilland_Comet

Sounds like a running joke at my work. Any hardware problem or design failure: you can fix it in software!

Do you work at Boeing? ;}

Bandaid fix if I've ever heard of one.

It's proper. This is exactly how you do things with life-critical embedded software. You detect when things are crazy, which of course shouldn't happen but nevertheless does, and you react in a way that is hopefully not disastrous.

The alternative is to ignore the problem.

Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad.

When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement?

Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.

The way I read it they are not detecting malfunctioning sensors, but essentially a computer fault caused by it's inability to deal with erroneous input. Rather than fixing the computer, they are essentially wrapping a great big try/catch around it....good God.

> are not detecting malfunctioning sensors

Because most of the time you can't. There is no reliable way of knowing when the sensor is lying to you.

You might try to reduce the probability of it happening and try to make it warn you (for example, redundancy, etc), but in aviation, even things that have a probability of failure of 10e-6 or even lower must be considered. 0% chance of reading an erroneous value doesn't exist.

The computer can deal with suspected erroneous input: it can ignore it. There. That's the standard way.

But what's erroneous input? A bit is flipped? A .1V increase over 0 makes it a logical 1, when it should be .3V? I mean, this is the domain of computer/electrical engineers in the first place, to prevent these kind of problems. This is what happens when you outsource your critical components to 3rd parties.

That interpretation seems like a stretch to me.

The way it's worded, that's exactly the interpretation that a lot of us in this thread get. If this is not actually the case, they should really word it differently.

This is the same Bloomberg that reported on the SuperMicro hardware "compromise."

I suspect your take is correct but I understand the alarm from the previous poster as the way it's worded makes it sound like they'll look just for condition X, Y and Z and _then_ take action versus your explanation which looks for a lot of incoming, bad data and then handles it.

This might just be a bad case of people talking past one another.

what happens when things are actually crazy and it causes a crash itself?

My jaw literally dropped. It actually took me a moment to realize I was sitting gape-mouthed reading that. We're talking airplanes here, not some LAMP stack.

If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.

We effectively have a front row seat to the sausage making of software development, its not pleasant.

That describes the whole 50+ years old 737 to it’s core.

It is.

But don't worry, when it is found out that it is a mess, we have a plan.

We can add a software patch to correct the bad software output. Not fix the bad software. But add some more to correct the output.

Genius!

its industry standard https://en.wikipedia.org/wiki/Application_firewall https://en.wikipedia.org/wiki/Web_application_firewall

To be fair, any hardware running a WAF in Prod these days is going to have more compute power than a 737.

Reply on news.ycombinator.com