FAA says computer failure was caused by contractors who introduced data errors
fortune.com
fortune.com
People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.
> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
That part doesn’t match my experience at all.
Good interview question to ask: “am I replacing a recent departure? If so, why?” You either get the truth or deception signal once hired. Both are helpful to know, and asking the question is free.
That happens. But not every time, and even "usually" requires some statistics collecting to be sustained.
Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll.
My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day.
I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.
That part doesn’t match my experience at all.
Good question though..
Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...
In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it needs to create robust solutions in the first place.
The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.
Probably not in this case, but you couldn't assume for every...
Maybe the CEO in the book gets his due?
Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.
TBH IDK the book was mentioned on HN and I figured I'd check it out.
Exactly! There wasn't any real fallout for Southwest, who had a shutdown less than a month ago, for similar outdated technology reasons.
There's running lean and then there is running recklessly.
I joke but the right amount of bureaucracy is good, but people are very careful about adding more of it because you can’t undo it
For example, running "/some/script > data-file" instead of "/some/script < data file" still fits the very generic wording the FAA is sharing publicly.
Could have been accidentally overwriting some idx/dat pair, or some raw database file, etc. Though, yeah, there should still be some controls, troubleshooting tools, logging, descriptive errors, etc, that would have made things more clear and recoverable.
It explains that procedures were circumvented and something in production was edited - something that was not allowed.
You can put as many procedures in place as you want, but unless the humans follow these procedures (I.e. not circumvent them deliberately) then someone always has some level of access where they can cause mayhem.
Sure you can programmatically enforce this to a large extent, but ultimately there are always some humans with prod access, or BGP editing access, or firewall write access, or access to that validation logic itself! And it requires that the humans follow the organisational procedures in place.
From what the article says, it sounds like the humans deliberately circumvented procedures, in which case, it is the engineers fault, and they should be disciplined.
It can be justified as "the instructions are out of date", "this is an urgent change", "just get it done!", etc.
Heck, sometimes following the written rules is derided as "working to rule" or "takes longer than others to complete the task".
I'm just saying that blame for these issues only sometimes belongs on the line workers.
In large org thinking, if something bad happens, and you add a rule, then it's fixed.
The problem is that this is not compatible with human psychology.
This leads to many rules, the rules are unstructured and impossible to learn, the rules probably also contradict each other.
In my experience, a rule without automatic enforcement should be the absolute last thing to depend on. If you do this, your org is broken.
The Toyota Production System with it's blame on process design instead of humans is still the best way to avoid these issues IMO.
Do 5-whys where human error is not allowed as a root cause. Design your process with poka yoke in mind. If somebody can forget something they will. Don't depend on it.
> Like other computer systems that are critical to operating flights, the FAA has imposed procedures to ensure data aren’t damaged by technicians working on them, said the person. The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
> Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
It is clearly implied that had the rules been followed, the corruption wouldn't have occurred.
Your blame-the-FAA scenario where the rules were incomplete or flawed isn't ruled out, but I don't see value in asserting it without evidence. I think it's perfectly plausible that someone just cheated a bit. That's hardly a rare situation in any IT environment, and so it seems like Occam favors the article's interpretation.
If that sort of thing actually happened a lot more C level people would have been fired for over-hiring, but that rarely happens.
I don't think this is a thing that actually happens.
But the previous CEO looks like he was the spreadsheet type and rejected these kinds of ideas. So who is to blame?
(Various elements and automations of the system have been created and updated, so these could have included improvements in design.)
> […] a data file was damaged as a result of a failure to follow government procedures, […]
The more critical processes humans are expected to perform, the worse the expected results.
That goes 10x for outside or temporary “contractors” who are not going to be experts in a systems unique, esoteric, or unexpectedly irresponsible points of fragility
Second, nothing would happen at a private company because they would involve marketing and marketing wouldn’t blame anyone in a press release. The only difference with FAA is that they didn’t let a marketing department write their release.
But really the real issue here is that everyone is posting to make wide generalizations from a one time event.
Can you share an example of where that has actually happened?
I doubt that. More likely: the defect would be identified and fixed, someone would start working on a training-wheels editor for the file to ensure the problem doesn't happen again. Any change in revenues or profits would be used by the CEO to drive change in whatever thing needs changing, and the whole incident would be forgotten. Source: have owned four private companies. If everyone is fired immediately, no one learns a lesson and everyone is terrified to give leaders bad news.
The engineering precautions in system design always involve trade-offs. There is no perfect system that could be created for acceptable costs. Ultimately it is always going to come down to humans following the rules, hopefully as few of those very-trustworthy humans as possible, though that's where the FAA may have been lax.
Sec. Buttigieg apparently agrees: https://www.nbcnews.com/news/us-news/software-blamed-faa-out...
> Transportation Secretary Pete Buttigieg told NBC News that he has asked the FAA, "to make sure that there are enough safeguards built into the system that this level of disruption can't happen because of an individual person’s decision or action or mistake."
When SpaceShipTwo broke up mid-flight, the FAA did not place all the blame on the pilot who accidentally engaged the feathering mechanism during ascent. They thoroughly dragged the aircraft's designers for producing a design that made it so easy to do such a thing in the first place.
They weren't willing to accept, "We said not to do it in the manual," as an appropriate safeguard in that case. Why are they doing it here?
If we want to call ourselves software engineers, we need to start holding ourselves and each other to higher standards. We can do better.
But I think the term "engineer" means more to you than it does most practitioners, unfortunately.
There's a limit to individual accountability. You need professional accreditation which brings training, standards, and legal resources.
Defaulting to red tape as the mechanism to improve the quality of software is also very short-sighted. Software engineering has the absolutely unique property among all engineering disciplines that we are working with abstract systems which are subject to formal verification. We can get arbitrarily high levels of assurance, with zero need whatsoever for some stodgy accreditation body.
This is not happening much at the moment because money is (artificially) cheap, so it makes sense to hack together some crap and get it out the door as quickly as possible. There are solutions to this; slapping regulations on software is not the best one.
Typical holier-than-thou software industry arrogance and hubris. There is nothing forcing red tape regulation on all engineers nor would it for software engineers.
It's absolutely bonkers that some companies don't trust the general population of devs with 5+ years of verified experience and still gives them technical interviews. But you can't really point them to a good standard to hire with either.
Only metrics resistant to Goodhart will work here, and most of them are illegal to use in the US under the current employment law paradigm. One of the reasons software as an industry makes so much money is there are still ways to implement Goodhart-resistant hiring in the US. Going into too much detail here is somewhat hazardous, so I will leave it there.
I don't exactly believe that.
And we have other professional exams/processes that generate good professionals. Folding over and saying "Oh we can't do this, Goodhart's law exists" just seems like a weak reason to me. https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good
These people had a public trust or a clearance. They've worked their way in the fed gov. I wouldn't immediately question malice or otherwise. You don't just throw away a career with sabotage.
Also having worked in a similar environment, I can guarantee you that although there may be a test env, it is nowhere near similar to prod. I've seen it myself with different firewall rules, different GPOs, different software hardening packages. And I also complained that there should be no distinctions between stage and prod - insomuch that DNS and SSL should be the only real differences.
I'm also acutely aware if my project has a similar incident, that I too would be scapegoated and fired. It's part of the course of this job.
We had dev and prod. No version control and no testing requirements. Was a while ago for sure (i used svn locally for sanity). But from what i hear not a lot has changed. This was not flight related software. They did have better tools and procedures but not by leaps and bounds
Ive had the displeasure of having another team write the Cloudformation scripts, and they wrote them in such a way that none could be redeployed, or changed. And since they're domain joined windows machines, effectively un-redeployable.
My first start, which died in situ, was to do the work to get stage up and running, and then clone stage into prod and changing the DNS and naming bits. Since this was a no-go, ended up piecemealing everything. It still stings too. We're nearly done and ready to go live as well. Ive been pushing for live since November. But when a single person speaks out, "delays get delayed by meetings to discuss in meetings to have meetings about delays" (note: I'm not making this up)...
I shouldn't complain too much. A good 4h a workday is filled with gaming and stuff, since the enforced delays and "hurry up and waits" require me to literally do nothing. But when I do work, I'm going at the 5-10x speed I'm accustomed to going. But when it takes weeks to open up SMB from internal machines to other internal machines... there's not a lot I can do to fix broken process.
And, I'm just a lousy contractor out of hundreds of thousands.
> The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
Can anyone here argue that, given arbitrary write access, intimate knowledge of the system, and the ability to disable any deployment and runtime checks I want, I wouldn’t be able to take out their production system?
I'm reminded of the punchline to an old joke:
"Doctor, it hurts when I do this!"
"Don't do that."
I just realized I have never disabled cell access on my watch.
It isn't clear to me why airplane mode is requested on airplanes. It doesn't reinforce crew control, it reinforces complete lack of crew control, i.e. crew control theatre.
What should they have done in this case that didn't involve flat out preventing it?
There has to be a limit to this reasoning, though. There is a threshold where an organization is so negligent that they can’t concentrate the blame on an individual, regardless of whether there was a policy violation.
It does suggest a big lapse in oversight: if this wasn’t the first time they did this, their security monitoring should have picked it up.
My guess is that this is going to be one of those situations where someone broke the rules in a way their immediate management knew about but sanctioned because the official process was dysfunctional. In those cases it’s really common to crucify the operator and ignore the larger problem.
If the take-away from this is “contractors should follow procedure and not make mistakes when entering data” then the FAA has not fixed anything.
So, yes, the system should have thrown some kind of "invalid magic number" or "unable to read header" error. Though that would have probably only helped a little. They were still slow to recover even after they understood the issue. Meaning they are missing still other things, like procedures to start over using fresh data. They had, for example, lots of partners that already had all the "known good data before the issue", but no way to take that and run.
But blockchains are specifically designed to keep on appending, no matter the mistakes of individuals, or even motivated sabotage attempts of sophisticated attackers.
Essentially, blockchains have a giant, full-coverage set of unit-tests in their shared & public consensus code – reviewed and running across many systems. And many of their designs have proven uptimes in adversarial environments for years.
And the task gets even easier when there's a small fixed set of established authorities with privileged keys.
Protecting those keys is also, of course, a hard problem. But compromised credentials is probably already a fatal threat for the legacy system(s) that were brought down by "contractors who introduced data errors". So a hypothetical modern crypto-consensus solution's failure modes remain a subset of those of the legacy system.
I am by no means an expert, however to me this system appears to include: a globally agreed upon data format, a transmission protocol hardware distributed across the planet in ground stations and air vehicles, and 70 years of legacy.
And all of that is enshrined in legislation across many countries.
And while there is a modernisation effort in progress, the nature of this system means that things will move slowly.
In 2023 anyone can suggest a better data format and distribution system if they start from scratch. Good luck getting the world to use it and good luck getting anyone to care.
The "globally agreed-upon data format" is essentially text blobs with a standard set of abbreviations, that assist search/filtering by flight-paths & planned waypoints. It seems to be a mere 150MB a year.
The inherently "move slowly" modernization effort can only be helped by a clearer outline of what's now possible. This is too important to prematurely dumb-down brainstorming ideas, based on low-expectations for the adoption process!
And a modern architecture for distributed updates & reliability could trivially mimic the old system's pilot-visible outputs, to remain familiar to tenured personnel.
Nor does distributed-consensus seem to apply in this case. For many datasets you want one source of truth (e.g. airport codes, approved abbreviations).
Crypto blockchains deal with very, very simplistic global states. Bitcoin, for example, has (a) 256 bits from the prior block, (b) a megabyte or so of transaction data. Every state machine update (mined block) totally replaces the prior state. And it crucially depends upon backtracking never being needed; for example, if a prior block's hash made it impossible for a new block to be created with enough zeroes in the hash (I don't think that's possible), then a global, manual fork would be needed.
And when something slips by, many eyes & many implementations make a consensus fix very fast. No one has 'fat-fingered' the major blockchains in a long time! Some proprietary/permissioned chains have had problems. And stuff as complicated as smart contracts can go awry in subtle ways. But NOTAM seems to be just a simple text log!
Distributed consensus inherently achieves one-source-of-truth without single-point-of-failure. As the notices relevant to individual flight-paths are inherently geo-local, with many independent submitters (like airports), any routing "through the FAA in DC" introduces risks – even if the FAA has final authority over who (which keys) can add entries.
Yes, Bitcoin's consensus state is relatively simple – but also far more compicated than NOTAM's even-simpler message. And continually redundantly verified & replicated by many independent actors & codebases, very successfully, for 10+ years.
Yes it can. That is not the question. The question is why do you think a bog standard, but correctly implemented and administered database could not handle the same?
Imagine the following: the task is to move two wheels of cheese 5 km from A to B. Your proposal is akin to someone comming and suggesting that a suborbital rocket can do the job. And you are right. It can. But so can a Toyota Corolla without breaking a sweat.
So tell me, why should we use a “hypothetical modern crypto-consensus solution” here, with all its complexity, instead of a well understood and bog standard and well implemented database?
But the centralized team running it could still "fat-finger" it. Any "master DB" system has problems that blockchain-like approaches have definitively solved, allowing progress on valid messages when large portions of the usual maintenance systems have gone down or been compromised.
Is there a major RDBMS out there that struggles with 150MB of data?
But if that's enough, why hasn't the FAA solved the problem already?
And even a "major RDBMS" is vulnerable to stodgy management leaving design in a state where it's vulnerable to a single central point of failure, where "2 contractors who introduced data errors" (!) can bring it down.
Modern highly-distributed consensus systems are not vulnerable to that. Neither accident nor malice from 2 government contractors could prevent all authorized updates worldwide in a distributed consensus system like Ethereum.
DC & northern Virginia (or wherever the FAA's systems hosted) could even spontaneously disappear into a lava pit and the official consensus set of NOTAMs would continue to update.
Hell, if you _really_ wanted to insist on an exchange data format, XML would work just fine.
The West has an addiction to foreign IT contractors. It’s a many hundreds of billions yearly industry globally, and is only possible because the largest and most critical institutions in our society refuse to invest in people to keep this knowledge in house and grow their own trained workforce.
LOL - let me show you Dubai
> When the system began having problems Tuesday night, technicians switched to a backup. But because the backup was attempting to access the same damaged data, it also didn’t work, the person said.
Their entire backup plan was to fail over to an identical secondary system that was supposedly redundant. But, it was clearly sharing the same database system as the primary, thus they didn't actually have a fully redundant system. They had 1 system with some parts of it having a secondary, i.e. not an actual DR / failover plan.
A corrupt file led to the FAA ground stoppage – also found in backup system - https://news.ycombinator.com/item?id=34348388 - Jan 2023 (388 comments)
US Halts Flights Nationwide After Key FAA System Goes Down - https://news.ycombinator.com/item?id=34337807 - Jan 2023 (36 comments)
FAA NOTAM System Outage - https://news.ycombinator.com/item?id=34337158 - Jan 2023 (214 comments)
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
Is there anyone there who heard about just culture, and root cause analysis? I can’t even believe they would publish this irresponsible stuff.
That's not a backup. Sounds like they have standby hardware for one tier of the architecture which they switched to, but since it wasn't a hardware failure it didn't change anything.
One hopes they review not only their processes which allowed (one assumes) a configuration change to be applied without being verified, but also their backup strategy...
People are remarkably failure-prone in the absence of controls and guardrails. Sadly, I suspect the individuals will get the rap instead of whatever crap process made it possible for the changes to pass muster and get into production without being caught and corrected.
Malicious changes to the IT systems were done outside the FAA by messing directly with the elections of officials who run federal agencies like the FAA.
Americans always look for a movie plot when the very real threats to “the homeland” looking to abuse their access are operating in the open and have been for many years… and they’re not low level software engineers.
https://twitter.com/Alwaleed_Talal/status/158597522656711065...
@Alwaleed_Talal Dear friend "Chief Twit" @elonmusk
Together all the way @Twitter
for instance, maybe that person has said something kind of evil, imo -- like maybe that an entire country should be ceded to some other country -- but even then i wouldn't want to overgeneralize -- i def don't like being associated with the most famous/rich americans when i go abroad, or interact online.
but i would agree generally that non-us investors buying up us politicians, pr firms, and tech and media companies is something we/us should be careful about.
On some level Americans are responsible for what our government and economic leaders do. We cant identify as and benefit from being Americans and then chide other groups for being racist when they complain about what our leaders do in our name. The reverse is equally true
the people that pushed, bragged and elbowed their way into executive management at a massive, eternally funded bureacracy are experts at disregarding the viewpoint, input and most of all directions from others
The first thing to do in such a position is to reward an inner loyal circle by increasing their benefits, authority and lessen the actual work, via contractors. This thread shows anecdotal evidence that setups involve far more than half the working headcount as contractors. The contractors do real work, take real blame, and are routinely pushed around since that is what people really do to each other.
At what stage of any of this do "we" take responsibility and make change?!? it is fortified income protection, not responsive systems, described here..
https://archive.org/download/SimpleSabotageFieldManualStrate...
"Any sufficiently advanced incompetence is indistinguishable from malice."
Fact remains that a system wherein this is possible is a system that has been exploited. Marshaling the forces to fix the system is the preferred outcome, unless you want the system to remain exploited (ie, you benefit from the situation).
I tend to find that idea is adjacent to conspiracy thinking.
If society is being “run” by anybody then either (a) they could do a much better job of it for themselves, or (b) they are very good at hiding.
“Powerful” people seem to often have the same problems as the rest of us: they seem to be on the receiving end of systemic issues they don’t understand as much as any of us, they often work long hours, they still get tragic family issues and chronic health issues.
https://thefederalist.com/2019/12/05/tucker-carlsons-critiqu...
Yes the wealthy mega-donors in American politics could absolutely do better by American workers than they are. How do these people being old and having health problems excuse their treatment of American workers?
A note is taped to the side of some filing cabinet that says
Remember to add a % with requsite spaces after the code you enter into the NOTAM file BEFORE saving file.https://www.cnsnews.com/commentary/hans-bader/faa-endangers-...
It could literally have been a lower case letter that didn't encode into whatever archaic language they use.
But: data errors introduced by a bad disk? In 2022? Storing your data on a hardware-backed (or, ZFS-backed, if you must) RAID6 (or equivalent) partition has been bog-standard for the past decade or so, and a partition is either online (and OK) or degraded (but still OK), or offline, at which point 'restoring the latest backup' (or just failing over to a replica) should bring you back to an OK state.
Then: apparently, restoring the offline partition caused bad data to be ingested, over and over, crashing the system again and again. Even if we pretend that solutions like SQLite have not been available for the past decade or so, 'skip past the head of the corrupted data file' has been best-practice since I last saw abominations requiring such hacks (say: Novell MHS), and sorry to repeat myself, over a decade ago?
TL;DR: please pay me US$ 200M, and I'll make sure all your problems are at least not of the kind solved a long time ago?