UK air traffic control meltdown
jameshaydon.github.io
jameshaydon.github.io
Short story: money. It costs money to do things well.
is this what refactoring means
Changes in how ATCs operate
Changes in how pilots operate
Changes in how airplanes receive these instructions (including the flight software itself, safety systems, etc.)
Changes in how airplanes are tested
Changes in how pilots are trained
Etc. In this case, the refactoring requires changes to hardware, software, training, manufacturing, and humans.
Functionally, I suppose it's the equivalent of upgrading an ID field that was originally declared as an unsigned 32 bit integer to a wider 64 bit representation. We may not be changing anything fundametal in the functionality, but every boundary interface, protocol, and storage mechanism must now suffer through a potentially painful modification.
The underlying flight plan standard were all created in an era of low memory machines, and when humans were expected to directly interpret data exactly as the programs represented it internally (because serialisation and deserialisation is expensive when you need every CPU cycle just run your core algorithms)
> additional data and heuristics
Directly mapping is impossible, so you cant just do a dumb ID-at-time pre-processing step (which is what your comment seems to suggest). You need a more complex pre-processing step that’s capable of understanding the surrounding context the identifier is being used in. A major issue with the flight planing system (as highlighted in the article) is that they attempted to do this heuristic mapping as part of their core processing step, and just assumed the ID wouldn’t be too ambiguous, and certainly wouldn’t repeat.
As for implementation complexity, you would hope they would use formal verification for something like this.
Especially since the implenting company is called out explicitly for failing to achieve this, and the risks of changing the well-established identifiers are also illustrated.
Perfect might be the enemy of the good then, or the standardization thing at least is a separate topic.
As a layman, I'd argue that such efforts would be band-aid and better spent on robust standardization
I.E. Yankee = YANKY. The pilot and ATC must be location aware. Apparently their software does not.
You need to be able to read, write, hear, and speak the identifier. (And receive/transmit in morse code)
Would it be okay to have an "area code prefix" in the identifier? Plausible (but practically speaking too late for that)
Talk to someone deep in the GIS rabbit hole and you'll get a rant about how bad W3W is: https://cybergibbons.com/security-2/why-what3words-is-not-su...
Von Braun reported that the rocket had six nines probability of success.
https://www.walklakes.co.uk/opus64534.html
The biggest fault (besides being proprietary) is that you must be online in order to use WTW. The times that you might need WTW are ALSO the times you are most likely to be unable to be online.
That doesn't seem to be the case anymore.
It's still not a great system – many included words are ambiguous (e.g. English singular and plural forms are both possible, and an "s" is notoriously difficult to hear over a bad phone line), and it's proprietary, as you already mentioned.
Aviation already uses phonetically pronounceable waypoint names. Typically 5 characters long for RNAV (GPS) waypoints, for example "ALTAM" or "COLLI". Easy to pronounce, easy to spell phonetically if needed, and easy to enter.
The problem is the list of waypoints are independently defined by each country, so duplicates are possible between countries.
Rather than replacing a system that mostly works (and mandating changes to aircraft navigation systems, ATC systems, and human training for marginal benefit)... an easier fix would just be to have ICAO mandate that these waypoints are globally unique.
If only.
For whatever reason, by convention navaids never use a country prefix. Even when it would make sense - the code for San Francisco International Airport is "KSFO", but the identifier for the colocated VOR-DME is just "SFO". (Sometimes this does make a big difference, when navaids are located off site - KCCR vs CCR for Concord Airport vs the off-site Concord VOR-DME, for example.)
It's even worse for NDB navaids, which are often just two letters.
Either way, we're stuck with it because it's baked into aircraft avionics and would be incredibly expensive to change at this point.
"Waypoint Charlie Alpha Dash ALKOG, over."
"This is an old machine, so just ALKOG then, over."
2. These are defined by the various regional aviation authorities. The US FAA will define one list, (and they'll be unique in the US) the EU will have one, (EASA?) etc.
The AA965 crash (1995-12-20) was due to an aliased waypoint name. Colombia had two waypoints with the same name within 150 nautical miles of each other. (the name was 'R') This was in violation of ICAO regulations from like the '70s.
you're saying what-3-words (W3W) is unsuitable for safety critical applications ? /s
But it's worse than that, there are confusables within small distances of each other:
https://cybergibbons.com/security-2/why-what3words-is-not-su...
Maybe they should release a new service called What4ActuallyVettedWordsAndWordCombinations ;)
That's ridiculous.
Something like FixPhrase seems better for use over radio.
or, or... hang with me here for a minute...
We could instead use one of these cool new hash algorithms that require a computer and use about fifteen thousand English words! I understand they are all the rage in the third world countries that lack a postal system.
I'm particularly a fan of the PGP word list (it would definitely require more than 3 words for this purpose, though) because it has built-in error detection (of transposition, insertion or deletion): Separate word lists are used for "even" and "odd" hex digits. This makes it, IMHO, fairly ideal for use over verbal channels. From the wiki: "The words were carefully chosen for their phonetic distinctiveness, using genetic algorithms to select lists of words that had optimum separations in phoneme space"
It sounds like the w3w folks did not do any such thing
EDIT: According to my napkin math, 6 PGP words should be enough to cover the 64 trillion coordinates that "what3words" covers, but with way better properties such as error detection and phonetic incongruity (and not only that, it is just over 4 times larger, which means it can achieve a resolution of 5 feet instead of 10)
As a New Zealander, the PGP list is unfriendly because there are plenty of words that are hard to spell, or are too US centric.
dogsled (contains silent d, and sleigh might be a British spelling)
Galveston (I've never heard of the place)
Geiger (easy to type i before e - unobvious)
Wichita (I would have guessed the spelling began with which or witch)
And why did the designers not make the words have some connection to the numbers e.g. there are 12 even and 12 odd words beginning with E - add 16 more E words and you could use E words for E0 to EF. Redundant encoding like that helps humans (and would help when scanning for errors or matches too)
I imagine it is even harder for ESOL people from other countries! I am sure the UI has completion to help - but I wouldn't recommend using that list for anything except a pure US audience.
It should be discussed like this! It's clear that the w3w people didn't even do the bare minimum here!
The thing is, once you agree that some words are subpar or need translations, you can do a 1-to-1 mapping.
The problem with What3Words is that supporting the original word set will always be a pain even if they release a v2 word set with a 1-to-1 mapping (I believe they've already released versions for other languages?)
re: Geiger- parsing it could trivially accept misspellings of words
Making a word list that could work well for speakers different English dialects and for speakers of English as a second language sounds really hard. Has such a list as been made?
Probably it is too hard so we will continue to ignore the problem.
They were too busy spending money on marketing, it's not like every news organisation ran a story about it by accident
Personally I could imagine that Maidenhead Locator System[3] may be more useful. It's just 4-12 chars (depending on what degree of accuracy you need)
[0] https://www.summerlandreview.com/news/bc-search-and-rescue-g...
[1] https://globalnews.ca/news/8258671/north-shore-rescue-what3w...
[2] https://www.squamishchief.com/local-news/squamish-search-and...
Oops. In software, the backup system should use different logic. When I worked at Boeing on the 757 stab trim system, there were two avionics computers attached to the wires to activate the trim. The attachment was through a comparator, that would shut off the authority of both boxes if they didn't agree.
The boxes were designed with:
1. different algorithms
2. different programming languages
3. different CPUs
4. code written by different teams with a firewall between them
The idea was that bugs from one box would not cause the other to fail in the same way.
A fix I would consider is to have the inputs more thoroughly checked for correctness before passing them on to the ATC system.
A B C D E
/
F G H I J
This might not be the reason why the developer chose to have the program essentially pull the fire alarm and go home in this case, but that's the impression I got.
I mentioned elsewhere that any ATC system has to be resilient enough to handle things like in-flight equipment failure, medical emergencies, and the diversion of multiple aircraft on account of bad weather or an incident which shuts down a major airport.
As for why the system "pulled the plug", the author of the article suspects that this particular error was regarded as something that would not occur unless something catastrophic had caused it, whereas, in reality, it affected only one flight and could probably have been easily worked around if the system had informed ATC which flight plan was causing the problem.
Thoroughly checking of the inputs as far as possible should be a given, but in this case, the inputs were correct: while the use of duplicate identifiers is considerably less than ideal, the constraints on where that was permitted meant that there was one deterministically unambiguous parsing of the flight plan, as demonstrated in the article. The proximate cause of the problem was not in the inputs, but how they were processed by the ATC system.
For the same reason, multiple implementations of the software would only have helped if a majority of the teams understood this issue and got it right. I recall a fairly influential paper in the '90s (IIRC) in which multiple independent implementations of a requirements specification were compared, and the finding was that the errors were quite strongly correlated - i.e. there was a tendency for the teams to make the same mistakes as each other.
It would be interesting to actually see the code.
All using communication links and protocols that have evolved organically since the 1950s, need global consensus (with hundreds of different countries' implementations), and which need to never fail.
The affected flight could have been vectored by ATC if needed to divert from filed FPL.
Way less work and a better doutcome than the “system throws hands in the air and becomes unresponsive”.
Air traffic control systems should at least be 2oo3[1] (3 systems independently developed of which 2 must concur at any given time) so that a failure of one system would still allow the other two to continue operation without impacting availability of the aviation industry.
Human backup is not possible because of human resourcing and complexity. ATC systems would need to be available to provide separation under IFR[2] and CVFR[3] conditions.
[1] https://en.wikipedia.org/wiki/Triple_modular_redundancy
[2] https://en.wikipedia.org/wiki/Instrument_flight_rules#Separa...
[3] https://en.wikipedia.org/wiki/Visual_flight_rules#Controlled...
They initially mounted two compasses in the cockpit, but Skarżyński taped one of them over so that it wasn't visible, saying wisely that if one fails, he will have no idea which one is correct.
Depends how it fails! For example, say, when you change direction one turns and the other doesn't.
But this was a 1oo1 system, and the human backup handled it well enough: a lot of people were inconvenienced, but there were no catastrophes, and (AFAIK) nothing that got close to being one.
As for the benefits of independent development: it might have helped, but the chances of this being so are probably not as much as one would have hoped if one thought programming errors are essentially random defects analogous to, say, weaknesses in a bundle of cables; I had a bit more to say about it here:
Heh, a hundred million pound outage. ;)
True, no-one seems to have died from it directly though.
It also serves as a reminder of how far we have to go before we can automate away the jobs of pilots and air traffic controllers.
I'll provide a relatively simple example.
Just even attempting to design a starfox game clone where the ship goes towards the mouse cursor using euler angles will almost immediately result in gimbol lock and your starfighter locking up tighter than unlubricated car engine going 100mph and unable to move. [0]
The standard solution in games(or at least what I used) has been to use quaternions [1] (Hamilton defined a quaternion as the quotient of two directed lines in a three-dimensional space,[3] or, equivalently, as the quotient of two vectors.) So you essentially dump your 3D coordinate into the 4D quaternion coordinate, apply your matrix rotations, then convert back to 3D space and apply your rotations/transforms.
This was literally just to get my little space ship to go where my mouse cursor was on the screen without it locking up.
So... yeah, I cannot even begin to imagine the complexity of what a Boeing 757 (let alone a 787) is doing under the hood to deal with reality and not causing it to brick up and fall out of the sky.
[0] https://math.stackexchange.com/questions/8980/euler-angles-a... [1] https://en.wikipedia.org/wiki/Quaternion
But regardless the more complex the code the worse idea it is to maintain three parallel implementations, if you won't/can't afford to do it properly
This is an artificial restraint. In the end, it comes down to risk management: "Are we willing to pay someone to make sure the system stays up when the computer does something unexpected?".
Considering this bug only showed up no, chances are there was a project manager who decided the risk would me extremely low and not worth spending another 200k or so of yearly operation expenses on.
In fact, make it adversarial testing such that this team is rewarded (may be financially) if mistakes or problems are found from the 1st team's program.
Nothing is perfect, though, and the pilot is the backup for failure of that system. I.e. turn off the stab trim system.
J. Gall
It seems like you should only failover if you know the problem is with the primary and not with the software itself. Failing over "just because" just reinforces the idea that they didn't have enough information exposed to really know what to do.
The bit that makes me feel a bit sick though is that they didn't have a method called "ValidateFlightPlan" that throws an error if for any reason it couldn't be parsed and that error could be handled in a really simple way. What programmer would look at a processor of external input and not think, "what do we do with bad input that makes it fall over?". I did something today for a simple message prompt since I can't guarantee that in all scenarios the data I need will be present/correct. Try/catch and a simple message to the user "Data could not be processed".
The real safety feature is the 4 hour lead time before manual processing becomes necessary.
One of the key safety controls in aviation is “if this breaks for any reason, what do we do”, not so much “how do we stop this breaking in the first place”.
1. Process controls: What do we do when this breaks for any reason.
2. Engineering controls: What can we do to keep this from breaking in the first place?
Both of them seem to be somewhat essential for a truly safe system.
1. You could know the solution, but it would be too heavy.
2. You could know the solution, but it would include more parts, each of which would need the same process on it, and the process might fail the same way
3. You miss something and it fails anyway, so your "what if this fails" path better be well rehearsed and executed.
Real engineering is facing the tradeoffs head on, not hand waving them away.
The process controls do however independently make things safe.
The reason for this is that there are 'unknown unknowns'—we accept that our knowledge and skills are imperfect, and there may be failures that occur which could have been eliminated with the proper engineering controls, but we, as imperfect beings and organisations, did not implement the engineering controls because we did not identify this possible failure mode.
There are also known errors, where the cost of implementing engineering controls may simply outweigh the benefits when adequate process controls are in place.
https://en.m.wikipedia.org/wiki/Hierarchy_of_hazard_controls
I mention this only because the Daily Mail headline pissed me off with it's usual bullshit foreigner fear mongering crap.
The software raised an exception because a "// TODO: this should never happen" case happened
A hardware fault would look like machines not talking to each other or corrupted data file unreadable
It's new code, from 2018 :) Quote from the report:
> An FPRSA sub-system has existed in NATS for many years and in 2018 the previous FPRSA sub- system was replaced with new hardware and software manufactured by Frequentis AG, one of the leading global ATC System providers.
Primary suffers integer overflow, fails. Secondary is identical, which also overflows. Angle of attack increases, boosters separate. Rocket goes boom.
Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire systemand take it out of production.
Airline industry code is a pile of badly-written legacy wrappers on top of legacy wrappers. (Mostly not including actual flight software on the aircraft. Mostly). The FPRSA-R system mentioned here is not a flight plan system, it's an ETL system. It's not coded to model or work with flight plans, it's just parsing data from system A, re-encoding it for system B, and failing hard if it it can't.
seems like poor engineering
The thing that blows my mind is that this was apparently the first time this situation had happened after 15 million records processed. I would have expected it to trigger much more often. It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.
This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software.
Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or lose their take-off slot.
Although there had been some “bus factor” planning and mitigation around this guy’s role, it involved engaging vendors etc. and would have likely resulted in a lot of disruption in the short term.
Failing the way NATS did means that all future flight plan data including for planes already in the sky are not longer being processed. The safer failure mode was definitely to flag this plan and surface to a human while continuing to process other plans.
1 https://news.sky.com/story/major-flights-disruption-caused-b...
The code that crashed had a bug, it couldn't deal with all valid data.
If you can stop the specific problematic plane taking off then keeping the system running is fine, but once you have a flight in the air it's a different game.
It's not totally unreasonable to say "we have an aircraft en route to enter UK airspace and we don't know when or where - stop planning more flights until we know where that plane is".
If you really can't handle the flight plan, I imagine a reasonable solution would be to somehow force the incoming plane to redirect and land before reaching the UK, until you can work out where it's actually going, but that's definitely something that needs to wait for manual intervention anyway.
Flight plans don't tell where the plane is. Where is this assumption coming from?
In this regard UK ATC can choose to do anything they like with a plane when it comes under their control - if they don't consider the flight plan to be valid or safe they can just instruct the plane to hold/divert/land etc.
I'm not sure the NATS system that failed has the ability to reject a given flight plan back upstream.
I have no idea what percent of the volume into the UK comes from outside radar control; if they asked a flight to divert, that may open multiple other cans of worms.
Any ATC system has to be resilient enough to handle a diversion on account of things like bad weather, mechanical failure or a medical emergency. In fact, I would think the diversion of one aircraft would be less of a problem than those caused by bad weather, and certainly less than the problem caused by this failure. Furthermore, I would guess that the mitigation would be just to manually direct the flight according to the accepted flight plan, as it was a completely valid one.
One of the many problems here is that they could not identify the problem-triggering flight plan for hours, and only with the assistance of the vendor's engineers. Another is that the system had immediately foreclosed on that option anyway, by shutting down.
That doesn't matter, because the previous unit actively hands the plane over. You don't need the flight plan for that.
What does matter is knowing what the plane is planning to do inside your airspace. That's why they're so interested in the UK part of the flight plan. Because if you don't give any other instructions, the plane will follow the filed routing. Making turns on its own, because the departing ATC unit cleared it for that route.
I thought practically, what's handed over is the CPL (current flight plan), which is essentially the flight plan as filed (FPL) plus any agreed-upon modifications to it?
> Because if you don't give any other instructions, the plane will follow the filed routing. Making turns on its own, because the departing ATC unit cleared it for that route.
Without voice or datalink clearance (i.e. the plane calling the new ATC), would the flight even be allowed to enter a new FIR?
Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed.
For these kinds of things the post mortem and remediation have to kinda take as given that eventually a not predictable in advance unhandled unknown error will occur and then work on how it could be handled better. Because of course the solution to a bug is to fix the bug, but the issue and the reason for the meltdown is a DR plan that couldn't be implemented in a reasonable timeframe. I don't care what programming practices, what style, what language, what tooling. Something of a similar caliber will happen again eventually with probability 1 even with the best coders.
How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the specific request instead of a complete shutdown every time a developer made a mistake.
A web server is a multi-user system, just like a country's air traffic control.
I mean, that's assuming the code isolating requests is also bug free. You just don't know.
> Having found an entry and exit point, with the latter being the duplicate and therefore geographically incorrect, the software could not extract a valid UK portion of flight plan between these two points.
It doesn't take much imagination to surmise that perhaps real world data is broken and sometimes you are handed data that doesn't have a valid UK portion of flight plan. Bugs can happen, yes, such as in this case where a valid flight plan was misinterpreted to be invalid, but gracefully dealing with the invalid plan should be a requirement.
I think there's a world of difference between writing bug free code, and writing code such that a bug in one system doesn't propagate to others. Obviously it's unreasonable to foresee every possible issue with a flight plan and handle each, but it's much more reasonable to foresee that there might be some issue with some flight plan at some point, and structure the code such that it doesn't assume an error-free flight plan, and the damage is contained. You can't make systems completely immune to failure, but you can make it so an arbitrarily large number of things have to all go wrong at the same time to get a catastrophic failure.
The flip side would be like a database failing to parse some part of its WAL log due to disk corruption and just said, "eh just delete those sections and move on."
The problem here is that one individual document failed to parse.
What surprised me more is that the amount of data existing for all waypoints on the globe is quite small, if I were to implement a feature that query by their names as an identifier the first thing I'd do is to check for duplicates in the dataset. Because if there are, I need to consider that condition in every place where I'd be querying a waypoint by a potential duplicate identifier.
I had that thought immediately when looking at flight plan format, noticed the short strings referring to waypoints, way before getting to the section where they point out the name collision issue.
Maybe I'm too used to work with absurd amounts of data (at least in comparison to this dataset), it's a constant part of my job to do some cursory data analysis to understand the parameters of the data I'm working with, what values can be duplicated or malformed, etc.
> The FPRSA-R system has bad failure modes
> All systems can malfunction, so the important thing is that they malfunction in a good way and that those responsible are prepared for malfunctions.
> A single flight plan caused a problem, and the entire FPRSA-R system crashed, which means no flight plans are being processed at all. If there is a problem with a single flight plan, it should be moved to a separate slower queue, for manual processing by humans. NATS acknowledges this in their "actions already undertaken or in progress":
>> The addition of specific message filters into the data flow between IFPS and FPRSA-R to filter out any flight plans that fit the conditions that caused the incident.
If a "this should never happen!" error occurs, then you don't know what's wrong with the system or how bad or far-reaching the effects are. Maybe it's like what happened here and you could have continued. Or maybe you're getting the error because the software has a catastrophic new bug that will silently corrupt all the other flight plans and get people killed. You don't know whether it is or isn't safe to continue, so you stop.
Because you eventually figure out that, yes, it does happen
What you're often led to is "if this happens, there's a bug in the code elsewhere" code. It's really hard to know what to do in that situation, other than terminate whatever unit of work you were trying to complete: the only thing you know for sure is that the software doesn't accurately model reality.
In this story, there obviously was a bug in the code. And the broken algorithm shouldn't have passed review. But even so, the safety critical aspect of the complete system wasn't compromised, and that part worked as specified -- I suspect the system behaviour under error conditions was mandated, and I dread to think what might have happened if the developers (the company, not individuals) were allowed to actually assume errors wouldn't happen and let the system continue unchecked.
That would have grounded only the plane with the flight plan that the UK system could not process.
Still a bug but with less effects to all the continent, because planes that could not get inside or outside the UK could not fly and that affected all of Europe and possibly more.
By the looks of it, it was few hours in the air by the time the system had a breakdown. Considering it didn't know what the problem was, it seems appropriate that it shut down. No planes collided, so the worst didn't happen.
In the case of a problem with all flights, the outcome would have been the same they eventually had.
Of course I have no idea if that would be a reasonable failure mode.
Yes. But also, it's an ATC system. Its primary purpose "is to prevent collisions..." [1].
If the system encounters a "this should never happen!" error, the correct move is to shut it down and ground air traffic. (The error shouldn't have happened in the first place. But the shutdown should have been more graceful.)
I didn't have to implement flight control software, but I had to write some stuff described by MIFID. It's a job from hell, if you take it seriously. It's a series of normative documents that explains how banks have to interact with each other which were published quicker than they could've been implemented (and therefore the date they had to take effect was rescheduled several times).
These documents aren't structured to answer every question a programmer might have. Sometimes the "interesting" information is close together. Sometimes you need to guess the keyword you need to search for to discover all the "interesting" parts... and it could be thousands of pages long.
Furthermore, identifiers from external systems always deserve scepticism. Even UUIDs can be suspect. Magic strings from hell even more so.
If programmer didn't know that repetitions are allowed, they wouldn't appear in the input to the fuzzer as well.
The mistake is too trivial to attribute it to the programmer incompetence / lack of attention. I'd bet my lunch it was because the spec is written in an incomprehensible language, is all over the place in a thousand pages PDF, and the particular aspect of repetition isn't covered in what looks like the main description of how paths are defined.
I've dealt with specs like that. It's most likely the error created by the lack of understanding of the details of the requirements than of anything else. No automatic testing technique would help here. More rigorous and systematic approach to requirement specification would probably help, but we have no tools and no processes to address that.
It totally would. The point of a fuzzer is to test the system with every technically possible input, to avoid bias and blind spots in the programmer's thinking.
Furthermore, assuming that no duplicates exist is a rather strong assumption that should always be questioned. Unless you know all about the business rules of an external system, you can't trust its data and can't assume much about its behavior.
Anyways, we are discussing about the wrong issue. Bugs happen, even halting the whole system can be justified, but the operators should have had an easier time figuring out what was actually going on, without the vendor having to pore through low-level logs.
I.e. this function wasn't the one which did all the job -- it already knew that the input was valid because the function that provided the input already ensured validation happened.
It's pointless to deliberately send invalid input to a function that expects (for a good reason) that the input is valid -- you will create a ton of worthless noise instead of looking for actual problems.
> Furthermore, assuming that no duplicates exist is a rather strong assumption that should always be questioned.
How do you even come up with this? Do you write your code in such a way that any time it pulls a value from a dictionary, you iterate over the dictionary keys to make sure that they are unique?... There are plenty of things that are meant to be unique by design. The function in question wasn't meant to check if the points were unique. For all we know, the function might have been designed to take a map and the data was lost even before this function started processing it...
You really need to try doing what you suggest before suggesting it.
> How do you even come up with this? Do you write your code in such a way that any time it pulls a value from a dictionary, you iterate over the dictionary keys to make sure that they are unique?
A dictionary in my program is under my control and I can be sure that the key is unique since... well, I know it's a dictionary. I have no such knowledge about data coming from external systems.
> There are plenty of things that are meant to be unique by design. The function in question wasn't meant to check if the points were unique. For all we know, the function might have been designed to take a map and the data was lost even before this function started processing it...
"Meant to be" and "actually are" can be very different things, and it's the responsibility of a programmer to establish the difference, or to at least ask pointed questions. Actually, the programmers did the correct thing by not sweeping this unexpected problem under the rug. The reaction was just a big drastic, and the system did not make it easy for the operators to find out what went wrong.
Edit: as we have seen, input can be valid, but still not be processable by our code. That not fine, but it's a fact of life since specs are often unclear or incomplete. Also, the rules can actually change without us noticing. In these cases, we should make it as easy as possible to figure out what went wrong.
That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processing the higher level queue".
I've actually seen this bug in different contexts before, and the lessons should always be: One bad job shouldn't crash the whole system. Error handling boundaries should be such that a bad job should be taken out of the queue and handled separately. If you don't do this (which really just entails being thoughtful when processing jobs about the types of errors that are specific to an individual job), I guarantee you'll have a bad time, just like these maintainers did.
Aviation is incredibly risk-averse, which is part of why it's one of the safest modes of travel that exists. I can't imagine any aviation administration in a developed country being OK with a "yeah just keep going" approach in this situation.
Flight plans don't contain any information relevant for collision avoidance. They only say when and where the plane is expected to be. There is not enough specificity to ensure no collisions. Things change all the time, from late departures, to diverting around bad weather. On 9/11 they didn't have every plane in the sky file a new flight plan carefully checked against every other...
They physically cannot be independent. The system works on an assumption that the flight was accepted and is valid, but it cannot place it. What if it accidentally schedules another flight in the same time and place?
Flight plans are not responsible for flight separation. It is not their job and nobody uses them for that.
As a first approximation they are used so ATC doesn’t need to ask every airplane every five minute “so flight ABC123 where do you want to go today?”
I’m staring to think that there is a need for a “falsehoods programers believe about aviation” article.
Maybe it can also produce the wrong segment within British airspace, meaning another flight plan might be processed successfully, but with the system believing it terminates somewhere it doesn't?
Maybe it's already been processing all the preceding flight plans wrongly, and this is just the first time when this error has occurred in a way that causes the algorithm to error?
Maybe someone's introduced an error in the code or the underlying waypoint mapping database and every flight plan that is coming into the system is being misinterpreted?
Considering their level 1 and 2 support techs couldn't access the so-called "low level" logs with the actual error message it's not clear to me they'd be able to keep up with a system with more complicated failure states. For example, they'd need to make sure that every plan rejected by the computer is routed to and handled by a human.
If from the code’s point of view it looked instead like a sanity failure in the underlying navigation waypoint database, aborting processing of flight plans makes a lot more sense.
Imagine the code is asking some repository of waypoints and routes ‘find me the waypoint where this route leaves UK airspace’; then it asks to find the route segment that incorporates that waypoint; then it asserts that that segment passes through UK airspace… if that assertion fails, that doesn’t look immediately like a problem with the flight plan but rather with the invariant assumptions built into the route data.
And of course in a sense it is potentially a fatal bug because this issue demonstrates that the assumptions the algorithm is making about the data are wrong and it is potentially capable of returning incorrect answers.
Coincidentally-identical waypoint names foxed UK air traffic control system - https://news.ycombinator.com/item?id=37430384 - Sept 2023 (64 comments)
UK air traffic control outage caused by bad data in flight plan - https://news.ycombinator.com/item?id=37402766 - Sept 2023 (20 comments)
NATS report into air traffic control incident details root cause and solution - https://news.ycombinator.com/item?id=37401864 - Sept 2023 (19 comments)
UK Air traffic control network crash - https://news.ycombinator.com/item?id=37292406 - Aug 2023 (23 comments)
- waypoint names used around the world are not unique
- as a sortof cludge, "In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced."
- but still you might get the same waypoint name used twice in a route to mean different places
- the software was not written with that possibilty in mind
- route did not compute
- threw 'critical exception' and entered 'maintenance mode' - i.e. crashed
- backup system took over, hit the same bug with the same bit of data, also crashed
- support people have a crap time
- it wasnt until they called the software supplier that they found the low level logs that revealed the cause of the problem
Kinda the opposite of a modern web/mobile app, complicated, massively bloated and breaks rather often :).
https://sfstandard.com/2023/02/02/sfs-market-street-subway-r...
You'll also be surprised to learn there's still parts of the US where there's no radar or radio coverage with ATC, if flying at lower altitudes. (Heck, there's still a part of the Pacific Ocean that doesn't have ATC service at any altitude.)
Aviation drove a lot of the early developments in networked computing, which also means there's some really old tech in the stack. The globally decentralized nature of it all and it being a life-critical system means it's expensive and complicated to upgrade. (And to be clear, it does get upgraded - but it in a backwards compatible way.) Today's ATC systems need to work with planes built in the 1950s, and talk to ATC units in small countries that still use ancient teletype systems and fax machines.
But yet it's all still incredibly safe, because the technology is there to augment human processes - not replace them. Even if all the technology fails, everything can still be done manually using pen and paper.
Essentially this is down to the lack of proper namespace, who'd have thought aerospace engineer need to study operating systems! I've a friend who's a retired air force pilot and graduated from Cranfield University, UK foremost post graduate institution for aerospace engineering with their own airport for teaching and research [1]. According to him he did study OS in Cranfield, and now I finally understand why.
Apparently based on the other comments, the standard for namespace is already available but currently it's not being used by the NATS/ATC, hopefully they've learnt their lessons and start using it for goodness sake. The top comment mentioned about the geofencing bug, but if NATS/ATC is using proper namespace, geofencing probably not necessary in the first place.
[1] Cranfield University:
Oh, this is typical in airline industry work. Ask programmers about a domain model or parsing, they give you blank stares. They love their validation code, and they love just giving up if something doesn't validate. It's all dumb data pipelines At no point is there code models the activities happening in the real world.
In no system is there a "flight plan" type that has any behavior associated with it or anything like a set of waypoint types. Any type found would be a struct of strings in C terms, passed around and parsed not once, but every time the struct member is accessed. As the article notes, "The programming style seems very imperative.".
If you've ever been a the airline desk waiting to check-in and an agent spends 10 minutes working with a passenger (passengers), it's because they got an ASD and the agent has to screw around directly in the the user-hostile SABRE interface to fix the reservation.
There are already too many problems with non-compliant or legacy (or just buggy) data emitters, with the complexity in semantics or timing of the interfaces, to try and be clever with badly formatted/encoded data.
It's already difficult (and costly) to make a system work as specified, so subtle variations to make it more tolerant to unspecificied behaviour is just asking for bugs (or for more expensive systems that don't clear the purchasing price bar).
You're right about all the buggy stuff out there, and that nobody wants to pay to make it better, though.
I feel the lack of investment in destroying the parsing- (and validation-) related classes of bugs is the worst oversight in the history of computing. We have the tools to build crash-proof parsers (spark, Frama-C, and custom model checked code generators such as recordflux) that - not being perfect in any way - if they had a tiny bit of the effort the security industry put in mending all the 'Postel's law' junk out there, we'd be working on other stuff.
I built, with an intern, an in-house bit-precise code generator for deserializers that can be proved absent of runtime errors, and am moving to semantics checks ('field X and field Y can only present together', or 'field Y must be greater or equal to the previous time field Y was present'). It's not that hard, compared to many other proof and safety/security endeavours.
Yes, but the code has to understand and model the input into a program representation: the AST. That's the essence of the "parse, don't validate" paradigm. Instead of looking at each piece of a blob of data in isolation to determine if it's a valid value, turn the input into a type-rich representation in the problem domain.
In the case of the FPRSA-R system in question, it does none of that. It's simply a gateway to translate data in format A to data in format B, like an ETL system. It's not looking at the input as a flight plan with waypoints, segments and routes.
Why the programmers chose to do the equivalent of bluescreening on one failed input, I can't say. As others have pointed out, the situation it gave up on isn't so rare: 1 in 15 million will happen. Of course switching to an identical backup system is a bad choice, too. In safety-critical work, there needs to be a different backup, much like the Backup Flight System in the space shuttle or the Abort Guidance System on the Apollo Lunar Module: a completely different set of avionics, programmed independently.
THAT SAID isolation of the decoding code and data structures, having a way back to either checkpoint/restore or wipe out bad state (or, proving the absence of side effects, as SPARK dataflow contracts allow, for example) is better design, I wish would be taught more often. I really dislike how often exception propagation is taught without showing the handling of side effects...
Of course bloody not. How is it a French airline's fault when it's a UK system? Systems like this should be foolproof with redundancies.
If one entry is bad reject it and carry on, even.
I'm not French and nor am I British. I feel neutral about both of them though I do live in the UK.
It's just logic, not nationalism. :/
I'm guessing it has something to do with how ICAO4444 is technically human readable, and how in some meaningful sense, pilots and ATC staff "prefer" it. e.g., maybe all ICAO4444 waypoints are "significant" to humans (like international airports), whereas ADEXP waypoints are often "insignificant" (local airports, or even locations without any runway at all).
Of course with 20/20 hindsight, it seems obviously incorrect to loop through the ICAO4444 waypoints in their entirety, instead of "resuming" from an advanced position. But why look at them at all?
Then it needs to find the ICAO part that corresponds, because the controller needs to use the ICAO plan that the pilot has.
If the controller sees other (auto generated) waypoints that the pilots don't have you get problems during operation. A simple example is that controllers can tell pilots to fly in a straight line to a specific point on their filed route (and do so quite often). The pilot is expected to continue the filed route from that point onwards.
They can also tell a pilot to fly direct to some random other point (this also happens but less often). The pilot is then not expected to pick up a route after that point.
The radio instruction for both is exactly the same, the only difference is whether the point is part of the planned route or not. So the controller needs to see the exact same route as the pilots have, not one with additional waypoints added by the IFPS system.
I had a cancelled flight recently and they booked people two weeks out because every flight from that day onward was full or nearly full. I showed up the next morning and was able to board the next flight because exactly one person had scanned in their boarding pass (was present at the airport) but did not show up for whatever reason to the airplane.
Beyond that, people just make alternate plans, whether it's taking a bus or taxi home, traveling elsewhere, picking another airline, anything is possible.
I work in logistics for a FMCG company and sometimes our main producer goes down and we run out of certain types of stock. We send as much out as we can and cancel the rest.
If they really want the stock the customers can rebook an order for tomorrow because they aren't getting it today. And we just start adding extra stock to each delivery.
It's the best of a bad situation.
We don't have the money to have extra trucks and very perishable stock laying about and I know the airlines don't pay 300 grand a month to lease a 737 just to have it sat about doing nothing. There's very little slack.
It is clear, even without any more information than that, it was a software failure (bad flight plan?)
It will be interesting to see if Frequentis has to pay a price for causing this
But the specifications allow ambiguities (non-unique waypoint ids) and the software did not handle this particular ambiguity correctly
An example are islands that serve few flights per week and can't justify heavy update investments.
Airplanes are generally spaced by hours and you need to do your math about where the airplanes are by hand. But again there's so little planes that risks are minimal.
Can't remember the name but I'm quite sure it belongs to Portugal.
There was a time recently when only 3 out of the 300+ air traffic control centers in the U.S. were fully staffed. All the rest were short-handed. Not sure how it stands today
Input is hard. Judging what constitutes 'input' in the first place can be harder.
A message switch [1] that I worked on had to deal with messages sources from 100's of different parties and while in principle everybody was working from the same spec (CCITT [2]) every day some malformed messages would land in the 'error' queue. Usually the problem was on the side of the sender, but sometimes (fortunately rarely) it wasn't and then the software would be improved to be able to handle that case correctly as well. Given the size of the specs and the many variations on the protocols it wasn't weird at all to see parties get confused. What's surprising is that it happens as rarely as it does.
The big takeaway here should be that even if something happens very rarely it should still not result in a massive cascade, the system should handle this gracefully.
What really is the heart of the matter is for the entire thing to be allowed to crash due to a problem with single transaction.
What you really want to do is to have firewalls. For example, you want a separate module that runs individual transactions and a separate shell that orchestrates everything but has no or very limited contact with the individual transactions. As bad as giving up on processing a single aircraft is, allowing the problem to cascade to entire system is way worse.
What's even more tragic about this monumental waste of resources is that the knowledge about how to do all of this is readily available. The aerospace and automotive industry have very high development standards along with people you can hire who know those standards and how to use them to write reliable software.
It all hinges on a whole bunch of assumptions and each and every one of those should be dealt with structurally rather than by patching things over.
Just from reading TFA I see a very long list of things that would need attention. Quick recap:
- validate all input
- ensure the system can never stall on any one record
- the system will occasionally come across malformed input which needs a process
- it won't be immediately clear whether the system or the input is at fault, which needs a process
- testing will need to take these scenarios into account
- negative tests will need to be created (such as: purposefully malformed input)
- attempts should be made to force the system into undefined states using malformed and well formed input
- a supervisor mechanism needs to be built into the system that checks overall system health
And probably many more besides. But this is what I gather from the article is what they'll need at a minimum. Typically once you start digging into what it would take to implement any of these you'll run into new things that also need fixing.
As for the last bit of your comment: I'm quite sure that those standards were in play for this particular piece of software, the question is whether or not they were properly applied and even then there are no guarantees against mistakes, they can and do happen. All that those standards manage to do is to reduce their frequency by catching the bulk of them. But some do slip through, and always will. Perfect software never is.
Maybe, but it must not be a common phrase because your comment is the first result when I search for it.
And it is also mentioned in this article: http://www.aero-news.net/subsite.cfm?do=main.textpost&id=ce2...
And that's about it? Do you have any other sources?
https://chaos.social/@russss/111048524540643971!
> Tonight we were wondering why nobody had identified the flight which caused the UK air traffic control crash so we worked it out. It was FBU (French Bee) 731 from LAX/KLAX to ORY/LFPO.
> It passed two waypoints called DVL on its expanded flight plan: Devil's Lake, Wisconsin, US, and Deauville, Normandy, FR (an intermediate on airway UN859).
> https://www.flightaware.com/live/flight/FBU731/history/20230...
> Credit to @marksteward and @benelsen for doing much of the legwork here.
> The software and system are not properly tested.
Followed by suggesting to do fuzzing tests.
* Automatically generating valid flight paths is somewhat hard (and you'd have to know which ones are valid because the system, apparently, is designed to also reject some paths). It's also possible that such a generator would generate valid but improbable flight paths. There's probably an astronomic number of possible flight paths, which makes exhaustive testing impossible, thus no guarantee that a "weird" path would've been found. The points through which the paths go seem to be somewhat dynamic (i.e. new airports aren't added every day, but in a life-span of such a system there will be probably a few added). More realistically some points on flight paths may be removed. Does the fuzzing have to account for possibilities of new / removed points?
* This particular functionality is probably buried deep inside other code with no direct or easy way to extricate it from its surrounding, and so would be very difficult to feed into a fuzzer. Which leads to the question of how much fuzzing should be done and at what level. Add to this that some testing methodologies insist on divorcing the testing from development as not to create an incentive for testers to automatically okay the output of development (as they would be sort of okaying their own work). This is not very common in places like Web, but is common in eg. medical equipment (is actually in the guidelines). So, if the developer simply didn't understand what the specification told them to do, then it's possible that external testing wasn't capable of reaching the problematic code-path, or was severely limited in its ability to hit it.
* In my experience with formats and standards like these it's often the case that the standard captures a lot of impossible or unrealistic cases, hopefully a superset of what's actually needed in practice. Flagging every way in which a program doesn't match the specification becomes useless or even counter-productive because developers become overloaded with bug reports most of which aren't really relevant. It's hard to identify the cases that are rare but plausible. The fact that the testers didn't find this defect on time is really just a function of how much time they have. And, really, the time we have to test any program can cover a tiny fraction of what's required to test a program exhaustively. So, you need to rely on heuristics and gut feeling.
It's true that fuzz testing is easiest when you can do it more at the unit level (fuzz this function implementing a core algorithm, say) but doing whole-system fuzz tests is perfectly fine too.
We literally talking about a parser shutting down an entire system rather than reporting malformed data.
Considering this is a "one in 15M cases" it seems to me that fuzzy testing would've caught this and probably more bugs in a short time span.
Similarly with a message queue, handling dead letters is textbook stuff, and you must have system tests to verify that poison pills do not break your queue.
I did not think the author was setting unreasonable expectations for the a priori testing regime. These are common best practices.
And if the functionality is "buried deep inside other code with no direct or easy way to extricate it from its surrounding" making it hard to test then that's just a further symptom of badly designed software in this case
In my mind they would only be approved once all involved countries review and process the plan. That way we don't need this ridiculous idea of failing safe on the whole uk airspace for a single error.
That day a single flight plan could have been rejected, perhaps just resubmitted and the bug quietly fixed in the background
The Daily Mail is a horrible, right-wing paper in the UK that blames 'foreigners' for everything. Particularly the French.
Out of curiosity, is there a corresponding French paper that blames the English or the British for everything?
Our right wing press is also desperately economically liberal so anything privately run is inherently better.
Maybe radio stations? Honestly, major respect to the daily mail for those snarky attacks that keep up the good spirits between our two countries.
It’s maybe the food or the weather that make them aggro ? Idk, but don’t worry, we love to hate the perfide Albion. Too.
Fellow French: am I wrong ? Maybe “valeur actuelle” could pull up that type of bullshit, but I think they are too busy blaming Islam to start thinking about our former colony across the channel.
There is really nothing to like or respect about the Daily Mail. https://www.globaljustice.org.uk/blog/2017/10/horrible-histo...
>our former colony across the channel
Touché! ;0)
But indeed, the unspoken rule is also that we hate them with a passion no matter what.
The one country that is often blamed for problems is rather Germany, but honestly even Germany doesn't get blamed for petty problems like that.
unrelated - this instantly caused me to think about tesla autopilot crashes that have been reported with emergency vehicles
[0] https://www.fastcompany.com/28121/they-write-right-stuff
But then it can also depend on what's in the contract and who will get the blame for the system functioning incorrectly. My guess here is that failing w/o attempting to recover was, while an overkill, a safer strategy than to let eg. two airplanes be scheduled for the same path (and potentially collide).
They had a four hour buffer until the world stopped, but most of that was pissed away because no one knew what the problem was.
This reminds me of: https://danluu.com/wat/
"Fly direct Quebec Xray Kilo Charlie Delta"
For example, Hungarian waypoints, at least the more important ones are normally named after cities, towns or other geographical locations near them, and use the locations name or abbreviated name, being careful that they can be pronounced reasonably easily for English speakers. Like: ERGOM (for the city Esztergom), ABONY (for the town Füzesabony), SOPRO (for Sopron), etc.
I think a lot of pilots and air traffic controllers would be irritated if they had to spend longer reading out clearances and instructions. In a world where vocal communication is still the primary method of air traffic control, there might be a measurable reduction in capacity in some busier regions.
Portsmouth, NH has a Sylvester/Tweety Bird approach: ITAWT, ITAWA, PUDYE, TTATT, followed by IDEED for the missed approach.
https://www.pilotsofamerica.com/community/threads/unique-way...
Australia has WALTZ, INGMA, TILDA, and also WONSA, JOLLY, SWAGY, CAMBS, BUIYA, BYLLA, BONGS
https://www.cntraveler.com/stories/2015-06-02/a-pilot-explai...
Disney has a whole lot of special fixes in Orlando and Anaheim. The PIGLT arrival passes through HKUNA, MTATA, JAZMN, JAFAR, RFIKI, TTIGR. I'm fairly sure I've heard about some variants on MICKY, MINEE, GOOFY, PLUTO, etc.
https://aerosavvy.com/wp-content/uploads/2016/04/MCO-PIGLT-S...
According to the same article, Louisville has LUUKE – IAMUR – FADDR.
This part stood out to me. I've found it super helpful to include a reference to which piece of days in working with in log messages and exceptions. It helps isolated problems so much faster.
A huge portion of "exploits" in the last 20 years have been "internal business APIs" if you will being exposed to malicious actors.
Formal verification or fuzzing could have helped them over that mistrust, but are not panaceas
1- the people writing and approving the specs even understand why this might be a good suggestion
2- the people ultimately approving the contract aren't in bed with the supplier
By contrast, its normal for an API to return 500 if something goes wrong and keep serving other requests. It would seem insane if it crashed out and completely stopped. Any idea why the parallel isn't true for a flight system?
Where does the "-R" come from?
No mention of plane, pilot, passenger and cargo manifests. So why the 4 hour lead time, is this the time it takes UK Authorities to look people up or workout if the cargo could be dangerous in an airborne Anthrax (Gruinard) Island [1] or Japanese subway Sarin [2], or an IRA favourite, fertilizer bomb thats bypassed the usual purchase reporting regulations used by people like Jeremy Clarkson and Harry Metcalfe as their store of wealth[3]?
It makes me wonder just how much more surveillance of the population exists, knowing I cant even step out of the front door without attracting surveillance of the type that followed Dr David Kelly.
Sure its not a cyber attack per se, carried out over the internet like a DDOS attack or a brute force password guessing attack with port knocking mitigation, but how would one carry out a cyber attack on this system if the only attack vector is from people submitting flight plans?
There sure is a constant playing down of the cyber attack angle to this which makes me think someone wants to Blurred Lines!
One point on the lack of uniquely named global way points, which is the main crux of the problem falling over if some are to be believed.
The USA demonstrates a disproportionate number of similar names, by virtue of Europeans migrating to the US [4]. So has this situation arisen with this system in other parts of the world like in the US? How can a country that created the globe spanning British Empire become so insular with regards to air travel in this way?
I'd agree with the initial assessment that there appears to be a lack of testing, but are the specifications simply not fit for purpose? I'm sure various pilots could speak out here, because some of the regulations require planes to be minimally distanced from each other when transiting across the UK.
On the point of ICAO and other bodies to eradicate non-unique waypoint names, its clear there is some legacy constraint still impeding the safety of air travellers, perhaps caused by poor audio quality analogue radio, so perhaps its time for the unambiguous and globally recognised What 3 Words form of location identifier, to come into effect?
The UK police already prefer it to speed up response times [4]. And although the same location can create 3 different words, suggesting drift with GPS [5], even if What 3 Words could not be used for a global system, having something a bit longer to create an easily recognisable human globally unique identifier is needed for these flight plans and perhaps maritime situations.
Obviously global coordination will be like herding cats, and if such a fixed size global network of cells were introduced, some area's like transiting over the Atlantic or Pacific could command bigger cells, but transiting over built up areas like London would require smaller sized identifiable cells. But IF ever there was a time for the New World Order to step up to the plate and assert itself, to create a Globally Unique Place ID (GUPID) for the whole planet, now is the time.
On the point of humans were kept safe, only by the sheer common sense of the pilots and traffic control tower staff, its not something NATS did or should claim, their systems were down, so everyone had to resort back to pen and paper and blocks in queues, and apart from Silverstone when the F1 British Grand Prix is on, is air space ever that densely populated.
NATS were caught with their pants down at so many levels of altitude, is this laissez faire UK management style that saw the Govt having to step in to bail out the banks during the financial crisis, still infecting other parts of UK life and still coming to light?
It's beginning to look a lot like Christmas!
[1] https://www.youtube.com/watch?v=_8Zr0IPtx80
[2] https://www.youtube.com/watch?v=RTr1lquCQMg
[3] https://youtu.be/LS54AJSadT4?t=279
[4] https://en.wikipedia.org/wiki/List_of_U.S._places_named_afte...
[5] https://www.bloomberg.com/news/articles/2019-03-21/u-k-polic...
[6] https://support.what3words.com/en/articles/2212837-why-do-i-...
To answer your question without conspiracy drivel, let's look up CAP 694: The UK Flight Planning Guide [0]
Chapter 1
> 6.1 The general ICAO requirement is that FPLs should be filed on the ground at least 60 minutes before clearance to start-up or taxi is requested. The "Estimated Off Block Time" (EOBT) is used as the planned departure time in flight planning, not the planned airborne time.
> 6.3 IFR flights on the North Atlantic and on routes subject to Air Traffic Flow Management, should be filed a minimum of 3 hours before EOBT (see Chapter 4).
Chapter 4
> 1.1 The UK is a participating State in the Integrated Initial Flight Plan Processing System (IFPS), which is an integral part of the Eurocontrol centralised Air Traffic Flow Management (ATFM) system.
> 4.1 FPLs should be filed a minimum of 3 hours before Estimated Off Block Time (EOBT) for North Atlantic flights and those subject to ATFM measures, and a minimum of 60 minutes before EOBT for all other flights.
So the answer is because the UK is part of a Europe-wide air traffic control system, which hands out full flight plans to all the relevant authorities for each airspace, and they decided 3 hours is needed so that all possible participants can get their shit together and tell you if they accept the plan or not.
An entirely separate system exists to share Advanced Passenger Information, i.e. passenger manifests [1], and it goes even further that airlines share your overall identity with each other, known as a Passenger Name Record [2], and a variety of countries, led by the USA, insist on this information in advance before the plane is allowed to take off [3]
If you're going to be paranoid, please work with known facts instead of speculating.
[0] https://publicapps.caa.co.uk/docs/33/CAP%20694.pdf
[1] https://en.wikipedia.org/wiki/Advance_Passenger_Information_...
[2] https://en.wikipedia.org/wiki/Passenger_name_record
[3] https://en.wikipedia.org/wiki/United_States%E2%80%93European...
Are their systems not joined up, or does the state just like making examples of peoples once on the destination country? I can watch this stuff happening to people at airport border controls on TV all the time so which is it? Their systems are not joined up or they just like making examples of people?
Here's a list of common reasons for the UK to refuse entry on arrival: https://www.gov.uk/government/publications/suitability-refus...
Some examples:
* Your valid travel document (as communicated digitally) turns out to be invalid when you actually present it
* You got a tourist visa, not a work visa, so why do you have work tools in your luggage?
* You turn up infected with Ebola
And is this a legal on the hook for returning passengers that are not allowed in the destination country?
I've got caught in the US when Hurricane Katrina was landing, and whilst we were flown out by our carrier, other carriers in this situation would also honour our ticket and fly us back to the UK.
It seemed like an exodus where all the carriers just got people out of the country as quickly as possible. We were on the last flight out of the airport, but this isnt a legal thing is it?
What exactly is the point of these support teams when they can't fix the most basic failure mode (a single bad input...)
To collect money on support contracts, I suspect.
Lesson learned for about the next ~5 years.
level 3 (devs) should have been brought in much quicker though
They bought a software from a third party and treat it as a "black box". There are few known ways that the software fails, and the local team has instructions on how to fix it. But if it fails in an unexpected way, good luck, it's impossible for the local team to identify and fix the problem without the vendor.
The reason it took so much was they realized too late that they need to call the vendor.
Probably you have to blame managers rather than engineers in the support team.
Which previous instances are you thinking about?
What it means is that on a regular basis, teams will be woken up at 2am because a batch process aborted on bad data; AND it doesn't tell you what data / where in the process it aborted.
The only possibility is to rerun the process with crippling traces, and then manually review the logs to find the issue, remove it, and then re-run the program again (hopefully remembering to remove the trace:).
Even when all goes per plan, this can at times take more than 4 hrs.
Now, we are not running a mission-critical real-time system like air traffic; and I'm in NO way saying any of this is good; but, it may not be the case that "two level of support teams didn't know anything" - the system could just be so poorly designed that with best operational experience and knowledge, it still took that long :-< .
On HN, we take certain level of modernity, logging, failure states, messaging, and restartability for granted; which may not be even remotely present on more niche or legacy system (again, NOT saying that's good; just indicating issue may be less with operational competence vs design). It's easy to judge from our external perspective, but we have no idea what was presented / available to support teams, and what their mandatory process is.
> Human lives were kept safe at all times
> The consequence of all this was not that any human lives were put in danger, ..
When you're arguing that cancelling 2000 flights cost £100M and that no human danger was incurred, something should feel off. That might be around 600k humans who weren't able to be where they felt they needed to be. Did they have somewhere safe to sleep? Did they have all the medications they needed with them? Did they have to miss a scheduled surgery? Could we try to measure the effect on their well-being in aggregate, using a metric other than the binary state of alive or facing imminent death? You get the idea.
Of course I agree with the version of the claim that says that no direct danger was caused from the point of view of the failing-safe system. But when you're designing a system, it ought to be part of your role to wonder where risk is going as you more stringently displace it from the singular system and source of risk that you maintain.
(We actaully have no major issues with the French at least in my generation, its all just good fun)
Edit: I lived in London for 3 years. I miss it every day.
Is that supposed to be a meaningful statement?
Yes, it can be argued that the software should've had more graceful failure modes and this shouldn't have thrown a critical exception. It can be argued that the programmers should've seen this possibility. We can argue a lot of things about this.
But the reality is that this is a mission-critical system. And for such systems, there're ways to mitigate all of these mistakes and allow the system to continue functioning.
The easiest (but least safe) one would be to have the secondary system loaded with code that does the same thing but written by a different team/vendor. It reduces the chance from 100% to much-much less that if any input provokes an unforseen, system-breaking bug in the primary, the same input will provoke the same bug in the secondary.
An even better solution is to have a triumvirate system, where all 3 have code written by different teams, and they always compare results. If 3 agree, great, if 2 agree, not so great but safe to assume that the bug is in the 1 not the 2 (but should throw an alert for the supervisors that the whole system is in a degraded mode where any further node failure is a showstopper), and if all disagree, grind everything to a halt because the world is ending, and let the humans handle it.
It can be refined even further. And it's not something new. So why wasn't this system implemented in such a way? (Aside from cost. I don't care about anyones cost-cutting incentives in mission-critical systems. Sorry capitalism...)
Capitalism is happy to have redundancy in mission critical systems all the time. Why would it care here?
A recent and very "in the face" example, also from the air travel industry would be the B737 Max and its AoA sensors. There were two, for two flight computers, but MCAS only used 1 flight computer and 1 AoA sensor, despite the already existing crosslinks between the flight computers and the sensors...
Pofit maxing first with the "no need for a new type rating for the pilots", then cost-cutting first in aeronautical engineering (solving an airframe design problem with software, plus designing a flight envelope protection system that can overpower the human pilots).
Then cost-cutting in software engineering and QC, rushing out software made by (probably) inexperienced in the field engineers and failing to properly test it and ensure that it had the needed redundancy.
Did you ever work with HA systems? Because this is how they work. It's two copies of the same system intended for the cases when eg. hardware fails, or network partitioning happens etc.
* Live + standby. Typically, the state of the live system is passively replicasted to the standby, where standby is meant to take over if it doesn't hear from the live one / the live one sends nonsense. (For example, you can use Kubernetes API server in this capacity).
* Consensus systems where each actor plays the same role, while there's an "elected" master which deals with synchronization of the system state. (For example, you can use Etcd).
In either case, it's the same program, but with a somewhat different state.
It doesn't make sense to make different programs to deal with this problem because you will have double the amount of bugs for no practical gains. It's a lot more likely that two different programs will fail to communicate to each other than one program communicating to its own replica. Also, if you believe you were right the first time: why would you make the other one different? You will definitely want to choose the better of the two and have copies of that than have a better and a worse work together...
EDIT: downvoting this question instead of responding is a pretty strange reaction.
Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.
In terms of how, it's really just a question of Schema A to Schema B mapping. Have a small team responsible for collection/organization of all the possible schemas and then another small team responsible for writing the mapping functions to transition existing data.
It would require will/force. Ideally, too, jobs of those responsible would be dependent on completion of the task so you couldn't just kick the can. You either do it and do it correctly or you're shopping your resume around.
However, you have to remember this is a global problem. You need to maintain 100% backwards compatibility with every country on the planet. So even if you upgrade your country's systems to something modern, you still have to support old analog communication links and industry standard data formats.
It’s invariably a complete clusterfuck.
[0] https://amadeus.com/en/insights/blog/celebrating-one-year-fu...
My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem of this scope/scale can be solved is by a team of people with extremely thick skin who are comfortable grabbing a beer and telling jokes after they spent the day telling each other to go f*ck themselves.
The amount of wise-cracks and beers your team can handle after a work day is not the determinate factor in success. /Most/ of these organizations /want/ to migrate these systems to something better. There is political will and budget to do so, these are still inglorious multi-decade slogs which cannot fail, ever, because failure means people die. No amount of attitude will change that.
Of course it isn't. But it's a starting point for building a team that can deal with what you describe (a decade-plus long timeline, zero room for failure, etc). If the people responsible are more or less insufferable, progress will be extremely difficult, irrespective of how talented they are.
Airplane logistics feels like one of the most complicated systems running today. A single airline has to track millions of entities: planes, parts, engineers, luggage, cargo, passengers, pilots, gate agents, maintenance schedules, etc. Most of which was created all before best-practices were a thing. Not only is the software complex, but there are probably millions of devices in the world expecting exactly format X and will never be upgraded.
I have no doubt that eventually the software will be Ship of Thesus-ed into something approaching sanity, but there are likely to be glaciers of tech debt which cannot be abstracted away in anything less than decades of work.
I would like to point out that the article (and the incident) does not relate to airline systems; it is to do with Eurocontrol and NATS and their respective commercial suppliers of software.
I am totally with you wishing these systems were more modern, having worked with them extensively, but I'm also realistic about the prospect. If every major airline regulator in the world worked on upgrading their ATC systems to something modern by 2023 standards, and everything went perfectly, we could expect to no longer need backwards compatibility with the old system sometime in 2050, and that's /very/ optimistic. These systems are basically why IBM is still in business, frankly.
The keyboards used by ATC don't even allow entering symbols: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F1...
(There is a modern replacement for AFTN called AMHS, which replaces analog phone lines with X.400 messages over IP... but the system still needs to be backwards compatible for ATC units still using analog links.)
Using a JSON format changes nothing. Just adds a few more characters to the text representation.
I agree with dundarius that "doing this in JSON" would not have changed the likelihood the bug could have manifested.
The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?"
Unless you're lying, you'd choose the former given the option.
No, the algorithm was "buggy" because it didn't account for the entry to and exit points from the UK to have the same designation because they're supposed to be geographically distant (they were 4000Nm apart!) and the UK ain't that big.
I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.
People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.
Yes, but in real life doesn't work. Processes have corner cases. As you said, people are lazy and will do everything to find the corner case to fit in.
Just an example from the banking sector. There are processes (and even laws) that force banks to use only certified, supported and regularly patched software: there are still a lot of Windows 2000 servers in their datacenters and will be there for many years.
But after you did it, you'd still have exactly the same problem. The cause was not related to deserialization. That part worked perfectly. The problem is the business logic that applied to the model after the message was parsed.
On top of that, RDF Turtle is the only widely used standard graph data format (as opposed to tree-based formats like JSON and XML). This allows you to reduce the hoop jumping when consuming responses from multiple APIs as graph union is a trivial operation, while n-way tree merging is not.
Finally, RDF Turtle promotes use of URIs as primary identifiers (the ones exposed to the API consumers) instead of primary keys, bespoke tokens, or UUIDs. Followig this rule makes all identifiers globally unique and dereferenceable (ie, the ID contains the necessary information on how to fetch the resource identified by a given ID).
P.S.: The problem at hand was caused by the algorithm that was processing the parsed data, not with the parsing per se. The only improvement a better data format like RDF Turtle would bring is that two different waypoints with the same label would have two different URI identifiers.
I'm not sure whether there is an existing RDF ontology for flight plans; it would probably be of low to medium complexity considering how powerful RDF is and the kind of global-scale users it already has.
I've had servers and software that I had never, ever used before stop working, and it took a lot less than four hours to figure out what went wrong. I've even dealt with situations where bad data caused a primary and secondary to both stop working, and I've had to learn how to back out that data and restart things.
Sure, hindsight is easy, but when you have two different systems halt while processing the same data, the list of possible causes shrinks tremendously.
The lack of competence in the "engineering teams" tells us lots about how horribly these supposedly critical systems are managed.
^1: they closed the West Drayton centre about twenty years ago; I don't imagine they moved their old IBM 9020D too, if they still had it by then. My comment is nonetheless only slightly exaggerated ;)
Additionally, perhaps he’s making the point that a language with an expressive type system makes solving problems like this trivial.