Hacker News Reader

Comment by lisper | Hacker News Reader

This article is very much at odds with my 15-year (1988-2000, 2001-2004) experience writing software at NASA (albeit at JPL, which is to say the unmanned space program, but still NASA). In my experience there was just as much politics, marketing hype, and general bullshit as I've seen in the commercial world. It was actually pretty amazing to me some times that things worked at all, let alone that they worked as well as they did. I think the real secret at NASA is not that they are better at writing code, but that they write it slowly (development cycles are measured in years) and they test the living shit out of it. Then, once it's working, they don't change it except under extreme pressure. But while it's being developed it's just as much of a sausage factory as anything else.

View on HN

As I wrote when this showed up on proggit,

Not to put too fine a point on it, but I would be very skeptical about using NASA as a good example for much of anything, but especially software development.

My bonifides: I'm 49; have spent roughly 25 years as a systems programmer, systems administrator, and general bit wrangler; and I worked for 8 years at the Marshall Space Flight Center, specifically the NASA Enterprise Application Competency Center.^1 (And yes, that's a thing. Started as IFMP (Integrated(?) Financial Management(?) Program), and was IEMP (Integrated(?) Enterprise Management(?) Program) when I started.) If you're a NASA employee, you might recognize STaRs (I worked on the rewrite, post Perl 1 and Monster.com), NPROP/Equipment, or DSPL/Disposal. And IdMAX, which I noped out of shortly after moving to the project.^2

NASA itself is a massively disfunctional organization, in my experience, and a failure to "cut through the bullshit" is a major reason why. For software development specifically, while I didn't do anything with "man-rated" development or the other important bits, I have strong doubts that they are any better than other avionics, automotive, or other embedded development organizations.

There was no mentoring. People tried, it didn't go over well, usually with the mentees.

You have to trust each other's potential, because there is no damn chance of getting any two projects to agree on anything. What goes on in that other silo is their business, not yours.

I did say, "I don't understand." A lot. Frequently pronounced "WTF?"

The list of "unreliable sources of knowledge" looks rather like a checklist of how things got done.

^1 NEACC is also the acronym for the North-East Alabama Community College, which I find ironic for no good reason.

^2 Unfortunately, I don't have a picture of me looking arrogant. Sorry.

> NASA itself is a massively disfunctional organization, in my experience, and a failure to "cut through the bullshit" is a major reason why.

I completely agree with this, though I think it's important to note two things. First, despite the dysfunction, they do, more often than not, make things that work under extremely challenging conditions. And second, a lot of the dysfunction is not their fault, but is a consequence of the fact that they are a government agency and hence ultimately answerable to Congress. Not only that, but they are a government agency created to fight a (cold) war that ended almost thirty years ago so they have been rudderless for a long time. The fact that they've done anything other than suck up taxpayer dollars is a testament to the incredible skill, both technical and political, of many of the people in the agency. Notwithstanding my vocal criticism of the organization, I have tremendous respect for many of the people who work there.

I will copy and paste my reply from your last thread:

NASA, like many large organizations known for prestigious accomplishments, can struggle to meet the needs of those doing the enterprise work (like internal HR apps as you mentioned). Especially in a place like the government, people's ego's feel small, and they try to compensate by making their projects bigger, grander, and overstated. This just results in bloat, too much talking, and BS as you mentioned.

I'm sorry you had that experience. If you ever consider coming back, I'd recommend getting involved with something more directly related to space applications.

I never knew what code post-80's NASA regulars actually wrote. When the MCC was moved off of the mainframe (MOC), it was all contractors. A crazy mix of skillsets in C/C++. A lot of subsystems/silos. Working across silos, it was incredible the brain trust you could meet from the old crowd (often retirees brought in for 2AM tests). I learned a lot of kids were afraid to say wtf. These old coots would love to talk for hours if you asked why something was done that way.

Considering the TRAJ assembly probably went back to DEW Line, I have no idea how they peeled that logic off. I still run into LockMart folks running ancient ATC code on Unix variants that have been out of service for over a decade.

Our code will never last that long.

I've also heard 1st hand stories and there have been multiple comments on HN regarding NASA.

The best I can tell is that the big difference in perception comes from people who worked on code for manned flights or anything that will interact with an actual human being vs non-manned and even non-mission critical code.

It seems that anything that will come close to a human is developed like it's the new arc of the covenant while for anything else it's YMMV based on the project and the team.

NASA has a classification of software into class A, class B, class C, etc. Class A is human spaceflight, class B is critical non-human spaceflight, and the rest is non-critical software.

The amount of process and oversight is highest at class A, and lowest at class E. If you are developing class E, you can do pretty much whatever you want and your team doesn't even need any training in software engineering.

This is completely true, in my experience.

If you want some fun, track down somebody working at the NASA Enterprise Applications Competency Center and ask them about the Office of Education software. (Me, I know nothing, through an exercise of good luck and skill. "No hablo ingles. Tengo hambre. Mucho muerte. <feigns death>")

I'm not sure about that. I worked at MSFC before and after Constellation was canceled and I'm not sure the perception isn't due to believing their own propaganda. From a short distance outside, it didn't seem to be running very well.

(I have a friend and excoworker there who was an aerospace engineer, specifically concerning the design of solid rocket motors, who explained why Constellation was not going to match its hype: you can't add a segment to a SSRB to get a bigger motor; it's a complete redesign anyway.)

To me, the point of career comparison is probably more for new programmers (e.g. assigning a mentor) rather than long term...though for a US Federal government job, fifteen years in not particularly long term. I mean the experience you describe would not be a surprising description of Google or Apple or IBM. Nor would it be surprisingly applied to the US Park Service or Harvard. Those features are applicable to pretty much every large institution.

Early in a career, can be a good time to try out large institutions because large institutions tend to have explicit structures for on-the-job training and career development. Early in a career is also a good time for some people to decide "big companies ain't for me" and for other people to decide "this is a good fit." Over time people in the startup ecosystem often become just as cynical as career government employees. I think it's more a function of age and experience and personality than anything else. Bright eyed and bushy tailed doesn't go with gray hair.

Not mentioned: Some software at NASA gets "human in the loop" testing. I work with GSFC, and that makes a lot of things difficult and very slow.

> ... they test the living shit out of it. Then, once it's working, they don't change it except under extreme pressure.

I wonder if self-driving car manufacturers will apply this same philosophy.

They're gonna have to. The first self-driving fatality (that isn't the fault of the driver or nearby drivers) is going to very severely impact any self-driving car manufacturer's brand.

Hasn't a Tesla self-driving system already caused a fatality when the system failed to identify a truck approaching from a perpendicular angle?

As far as I know, all investigations into Tesla self-driving fatalities didn't find the system at fault.

The author is young, and has been there for 3 years. I'm guessing this is anecdotal, and he happens to have had a good experience. Always possible things have improved though?

The fundamental truth that must be considered of any article like this is "YMMV". (Your mileage may vary)

Especially in a big organization, things are different depending on the division you're in and where you work. I work at a large .gov institution and am only here a decade later because my leadership team in the early days were great bosses and amazing mentors. They taught me a lot both in a professional and personal level. I aspire and in many ways fail to be the same.

I look at other parts of the organization and see great places as well divisions where employees are treated like crap and are demoralized to the point of depression.

Working in a "good part" of a big company or institution can be amazing.

This is one of the best-chosen comments in the thread. The OP has encouraged commenters here to make generalizations about "NASA" software, but this does not make sense. It's too big, too dispersed, and the software developed is in a big range from research codes to operational codes to human-rated (see upthread: https://news.ycombinator.com/item?id=14886727).

In short, s/w practices and the dedication and capabilities of the people writing the software vary too much to characterize at that level of granularity.

I don't think you contradict each other. You just seem to focus on different sides of NASA development process.

Does that mean that as long as you test the shit out of it, it doesn't matter how it gets done?

No, it still matters. But exhaustive testing can produce reliable products from unreliable processes. The extreme example of this is evolution itself.

Pretty sure that if repeated tests reveal aspects of the program to be deficient, the "how it was coded" will come out the proverbial wash.

That seems a reasonable position - yes.

Reasonable if you ignore maintainability and standards compliance.

Not a coder, but if you used an idiom that worked because of a compiler quirk then that could lead to problems down the line but related to maintenance per se, hence why I added the second part.

I doubt NASA would ever ever upgrade compilers on software once tested. If it is like a aviation, there might even be manual passes over the assembly code for further verification.

I was thinking in a historic archiving sort of situation, bringing to mind recent stories of NASA code being unearthed. Basically I was expecting "it passes all the current unit tests" to be an insufficient standard for NASA coding.

In any mission critical situation the entire tool chain is just as locked down and controlled as the "source code," even down to the silicon.

the ISS laptops run on commodity hardware and software, often undocumented, and from my interview with that team the notion that they have some fully deterministic system set up for their operations is horrendously false, and they've run into many issues because of it

As an aside, is JPL really NASA? Whatever happened with the lawsuit about HSPD-12 badges?

JPL's status is unique. It is technically an FFRDC (federall-funded research and development center) whose money comes from NASA. It is considered a NASA center, but it is unique among NASA centers in that its employees are Caltech employees, not federal government employees. That's why I added the prominent disclaimer. Whether or not it's "really" NASA is somewhat in the eye of the beholder. But they are definitely not a private enterprise like Space-X.

If a lot of process is applied to verify the robustness of the code, then to me the overall development process is not a sausage factory.

> This article is very much at odds with my 15-year (1988-2000, 2001-2004) experience writing software at NASA (albeit at JPL, which is to say the unmanned space program, but still NASA). In my experience there was just as much politics, marketing hype, and general bullshit as I've seen in the commercial world.

An institution's philosophy is almost always bullshit. Whether it is Google's "Don't be evil" or north korea's "worker's paradise" or our own "capitalist democratic paradise".

The media isn't a fount of journalist integrity. The government isn't a found of "goodwill towards its citizens". The military isn't "defending the peace".

It's sad but the more you work and the older you get, the more the facade crumbles. At the end of the day, it's just fallible humans trying their best to get by.

Reply on news.ycombinator.com