Boeing 787s must be rebooted every 51 days to prevent 'misleading data' (2020)
theregister.com
theregister.com
Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.
People have been writing HFT systems in GC languages (Java, OCaml) for a while, and as you mentioned, you need to know when to pause/unpause/prevent collection in the critical section.
This [1] is an excerpt from a talk with Dave Lauer [1], who has done it in Java - it took his system ~40 μs from receiving data to producing market order, and he explains in great detail what amount of work the system had to do. And this was >10 years ago.
(from my experience anyway in writing critical-path allocation-free C# with Unity).
That's his point. It's like saying humans can walk over a lake, as long as it's dry season and the lake is now 10 inches deep.
You can use garbage collected languages as long as you don't collect. (!?!?!)
I'm even more hardline. IMO everything time or life-critical should have static memory allocation, period.
The surprising thing here is that it is considered worth it with those limitations.
My hope/guess is that because of the GC hype there wasn't any good alternatives.
With the alternatives gaining maturity now I'd hope that that decision wouldn't be made today (when starting from a clean slate).
These people make decisions based on Excel spreadsheets that are not properly documented. I wouldn't trust their judgement, despite the fact they managed not to crash the global financial system too frequently.
It shouldn't be that surprising. Languages like Java and C# are perfectly sensible languages for writing the vast majority of a lot of trading systems. So it's eminently reasonable to use them, or equivalent, and deal with the bits that are GC-phobic as special cases.
I've worked with/on trading systems written is mainstream languages including C++, Java, C# and Python, as well as less mainstream such as APL and Smalltalk. I can safely say that there are way more critical concerns than whether or not it uses a GC.
>GC just increases your energy bill and helps heat up the planet.
Were C++ compilation times that use 100% of my PC taken into consideration?
And not just for final binary, but also the compilations that happen everyday for development purpose too
How? GC does its work in large batches once in a while, whereas for example Arc (the only serious alternative to what GC brings, which is the ability to have arbitrary graphs of objects) does its work much more frequently, with additional effort expended on atomic operations in a multithreaded environment, and with additional memory writes to boot. Somehow I doubt that GC "just increases your energy bill".
In retrospect I think I should have put more effort into breaking the feature factory culture than the discipline culture. It's hard to get clean if your drug of choice is ready at hand.
https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
_________
From: k...@rational.com (Kent Mitchell)
Subject: Re: Does memory leak?
Date: 1995/03/31
Norman H. Cohen (nco...@watson.ibm.com) wrote:
: The only programs I know of with deliberate memory leaks are those whose
: executions are short enough, and whose target machines have enough
: virtual memory space, that running out of memory is not a concern.
: (This class of programs includes many student programming exercises and
: some simple applets and utilities; it includes few if any embedded or
: safety-critical programs.)
This sparked an interesting memory for me. I was once working with a
customer who was producing on-board software for a missile. In my analysis
of the code, I pointed out that they had a number of problems with storage
leaks. Imagine my surprise when the customers chief software engineer said
"Of course it leaks". He went on to point out that they had calculated the
amount of memory the application would leak in the total possible flight time
for the missile and then doubled that number. They added this much
additional memory to the hardware to "support" the leaks. Since the missile
will explode when it hits its target or at the end of its flight, the
ultimate in garbage collection is performed without programmer intervention.
--
Kent Mitchell | One possible reason that things aren't
Technical Consultant | going according to plan is .....
Rational Software Corporation | that there never was a plan!
When the point of the product is to kill someone, you can't just stochastically measure shit like this. If the device is ever out of control before it ceases operations, you're looking at Geneva Convention level offenses.
Even software designed to save lives can't get away with this sort of thing.
It just drops?
https://www.scienceabc.com/innovation/why-do-some-missiles-e...
Apparently it will "hijack" more memory and attempt to reconstruct it's orders.
But, real world, I would say that data corruption would occur and outside of chemical/physics, it will just drop and lockup.
May have a failsafe to detonate, but that is a fun question indeed.
If you think getting a license is the way to fix quality issues, I have a bridge to sell you.
That's a joke. Russia is intentionally bombing Ukrainian civilians, you think they're going to have a single War Crimes charge put against them? How many civilians did the US unintentionally kill across Afghanistan and Iraq, hundreds of thousands? See any charges there either?
I don't know how I could possibly get one but I'd love to read an independent comparison of this sort of behaviour between US operations and Russian ones. My belief is that the US aims not to kill civilians but is often careless and nets a lot of collateral damage, whereas Russia doesn't care at all and will happily bomb schools if it thinks there's a target in there. But of course I mostly read Western media and writing so my view is potentially very biased.
This sort of analysis is completely normal in software. E.g. in safety critical software you often analyse maximum possible stack depth to check that your stack is big enough (one of the reasons why recursive code is sometimes disallowed). This is exactly the same class of analysis.
Their problem was that when they operated their device it was indeed a single device, but later became many more. If they paid for the extra licenses they didn't need it would be ripping off the government; if they failed to pay for the extra license at some point they would be in breach of their contract with the government as all these unlicensed devices were released.
It rapidly became obvious that their application was a MIRV and that they weren't really going to change but had asked us to talk with them as part of their process of negotiating a deal with their current supplier.
It's hard to imagine really worrying about your license agreements if there actually were a global thermonuclear war.
In our debug build, I added a registry key that would start the concept of time at 49-ish days, so that we could test that everything was working without waiting 50 days. However, before we shipped, the intrepid test team did actually run the encoder for 50 days to prove that it worked. (On Windows NT, of course. Windows 95 with good hardware and drivers was perhaps more stable than its reputation, but still...) Though TBH we definitely wouldn't have delayed a release by 49 days if it had failed at the last minute.
https://blogs.windriver.com/wind_river_blog/2011/09/boeing-7...
"The functions of VxWorks have nothing to do with the data issue you are highlighting in the 787," a spokesperson added. We are happy to clarify our coverage.
I would hope the advisory leaves some room for error in the reboot interval, though.
It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
And in fairness, reliability is a major point of automation.
Whereas going decades without seeing a qualified doctor has been the default state since the dawn of homo sapiens with nothing more than a good night’s sleep.
I would not compare human brains to tech. Maybe you can call them simpler and thus more reliable. The human brain is processing far more data than any processor humanity has made. The only comparison is humans handle various types of data whereas computers are designed to handle just a small batch.
Airplanes are not reliable. That's why they require frequent and regular maintenance.
Ignoring the poor analogy linking brains/humans to human built systems, if every human maintained their mental and physical health as much as airplanes get, then they'd all be much better off.
Load the airplane's "brain" with algorithms for how to survive, cooperate, adapt and replicate under every conceivable condition and you'll end up with a comparable system more complicated and worse-performing than ourselves. Not really a fair comparison.
Self-driving cars (and robotics) would be better examples, since they attempt to emulate a broader swath of our sensory capabilities.
Later Hominids are meant for this. Without being able to pass long time chasing our prey to the moment of its exhaustion, we'd not have been able to secure the cholesterol needed to further evolve our brains.
Small fish/shrimp can appear to "teleport" since they dart away so quickly, but if you harass them enough, you exhaust them into a state of shock. It's no longer a chase, it's a harvest.
Even in manhunting, you get nowhere chasing the subject since you move at the same speed and the element of uncertainty tilts in their favor. So you try to anticipate where they're going and ambush them there. You turn the chase into a trap.
We're apex predators for a reason. Exhaustion chases don't make sense.
The hotter the weather, the easier this gets.
I think it has to do with the way the human visual system effectively does edge detection and then groups those into objects and attaches meaning. When you are extremely sleep deprived, the edges gets grouped differently and you start seeing things that aren't there.
I could judge what was real or not based on likelihood of me seeing said things in the wilderness, or if I got close enough I would be able to tell what I was really looking at. Another runner later informed me that I had told him he wasn't real when he passed me.
I once saw death (exactly as most of us imagine it, dark robes and a scythe) while driving on a highway late at night, I understood the message my brain was sending me: I just took the next exit and slept for an hour (it was too cold to sleep longer, but it made a big difference on my concentration anyway).
It doesn't even feel that bad once you get past the slump in the 20-24 h region.
The answer, of course, is "a computer."
Now I'm curious: how long does it take to reboot Boeing 787.
https://ad.easa.europa.eu/blob/2020-06-14.pdf/AD_US-2020-06-...
Edit: patent pending
Of course since I'm still here, it wasn't it. After a while I felt the vibration of the engines again and was relieved. Maybe the pilots had to reboot the plane.
So, to me, the power cycle thing is not super surprising. That fact that the FAA felt the need to order Boeing to do it seems a bit suspect though. That's probably the real story.
I don’t know how long rebooting a 787 takes, but turn Around Time of an airliner may be too short to do a reboot. http://aviationavi.com/turn-around-time-tat/:
Ryanair a private ULCC of Europe and one of the largest ULCC of the world having a TAT of 25 minutes does not have a seatback pocket.
Wondering why?
Back in 2004, removal of seat back pockets greatly reduced the time for which the Ryanair aircraft stayed on the ground. That’s is because it takes more time to clean an airplane with seatback pockets.
So, that’s 25 minutes from touching down to being airborne again. Chances are the plane is standing still for less than 15 minutes. Long-haul flights take longer (up to two hours, according to https://simpleflying.com/aircraft-turnaround-process/)
Also: it wouldn’t surprise me if you could not refuel an airliner while it is rebooting, all lights may be off for a while, etc.
This simply does not and cannot happen to an airline. If it does people will be fired and processes revised.
with the current state of Boeing, nothing would surprise me, and there's very little i would accept at face value from them.
Maybe if there's a really badly run airline it might be a problem, but there are much more severe problems than software in a case like that.
Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
Yes, like the mundane software overriding pilot's intent from software written specifically to not retrain pilots on the system? Yup, sounds mundane to me.
When I said I don't trust anything from Boeing, it had little to do with this 50 day reboot, but much more in line with the Boeing is allowed to self certify and has been shown to not do that in a good way. They have ruined that trust because of greed. You might think that's a broad brush? At this point, every little bit is just another instance of death from a thousand cuts.
On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
I would honestly hope that would be true about any airplane from any manufacture. Fly-by-wire makes some of that more difficult when the wires go through the computer, but even then I'd have assumed some sort of 100% manual capability. Like when Y2K nutters were predicting airplanes falling out of the sky because of 2-digit years rather than the logical the planes would behave according to laws of physics for an intact air frame.
Some discussion from 6 years ago:
The engineers behind Boeing planes have built an incredibly sophisticated machine which omitting a very nessescary but not so inconvient task of rebooting every n days, performs its primary directive well.
One bug, does not imply two bugs.