Hacker News Reader

Comment by trollied | Hacker News Reader

Has anything happened regarding IT staffing at the FAA recently? There was the NOTAM outage last week [1], and now this!

[1] https://news.ycombinator.com/item?id=34337158

View on HN

The NOTAM system was apparently running on old SPARC hardware so I’d easily believe that they haven’t been given enough budget to hire engineers and are basically keeping the lights on. One peril of the government funding model is that it’s effective at shoveling money to contracting companies but not having skilled oversight, so you tend to see lots of big projects which founder under their own weight.

NOTAM could be run from a pre-historic Toshiba mainframe. it's static. Doesn't need billions in funding.

It's static, until a contractor ignores the post-it note on their monitor that says, "IMPORTANT: ALWAYS add an extra newline BEFORE saving the file".

Once that happens, you have to hope that your backups work - why bother having disaster recovery drills for a static system?

Once you realize that your backups are hosed, you may experience a creeping suspicion that software is never truly static when human operators are involved.

It doesn’t need billions but it needs enough for dedicated staffing. Many federal agencies have the problem that they’re not given general funding for staff but are given money for large contract projects. It’s not uncommon for that to be structured in a way that, say, a big project has rooms full of developers, PMs, testers, etc. but something small has like one dude keeping it going when they can find time and a stream of unanswered requests to management asking go either fund or decommission it, hire a replacement for people who retired, etc. or that the big project which was supposed to replace it is years overdue.

I would not be surprised to learn that this was the case here, too.

I would love to see more information about the underlying architecture of the NOTAM system, if you can recall where you saw that.

Not sure how reliable this guy is but it’s certainly easy to believe:

https://twitter.com/PatrickMoorhead/status/16139375120027729...

SPARC gear would last that long though, I recall playing with a T1 system and opening the chassis to watch the Christmas tree of self check lights on every part of the board turn on. The entire system was designed to self check every component down to like, individual VRMs.

Yeah, that's both good and bad — it's great that it's reliable but that's also the kind of thing I could see people getting complacent about because, hey, it hasn't failed before.

I don't know anything about these two incidents. But, FWIW, some of the past professional work I'm most proud of was on FAA contracts (on other systems). I'm sure they're handling the immediate situation, and -- consistent with their culture -- they'll analyze what went wrong and can be learned, and follow through on that.

Maybe there's an ongoing cyber attack ? :-)

That isn’t being admitted to?

Happens more frequently than you think.

Admitting attacks can lead to more attacks if you’re not prepared

Nope, just the regular old spending money on new projects that benefit the careers of powerful politicians but refusing to spend on maintenance attack.

Probably just more visibility on the matter. Once a major issue happens, things tend to get followed for a while. It can get bandwagoney

For what it's worth, the METAR network (NADIN) is maintained by the NWS, not the FAA.

(Following up on this, I was actually mistaken. I believe NADIN is actually an FAA network.)

You mean besides the last three years of reduced on-site staffing, and employee travel to support maintenance and modernization? No.

$5 says this is the XKCD for this. Corrupted DB file was the culprit, as announced by the FAA. It's a system that relies on user inputs, often manual, and it's all being put into a database. My bet is a user decided to put in a bunch of "fun characters" to make their input easier to read. You can't account for and sanitize all levels of stupidity on user inputs.

https://xkcd.com/327/

> You can't account for and sanitize all levels of stupidity on user inputs.

That is false and a defeatist attitude.

Sanitization is not the right solution anyway. If you are working with any form of a database and don't know from the top of your head how to avoid query injection attacks then you should look up in the manual of your database. The solution is most often called "parameterized query" or something similar.

Of course you can account for that. They're just byte strings. That xkcd is poking fun at incompetent programmers, not stupid users.

Reply on news.ycombinator.com