The Loyal Engineers Steering NASA’s Voyager Probes Across the Universe | Hacker News Reader

The Loyal Engineers Steering NASA’s Voyager Probes Across the Universe

mobile.nytimes.com

307 pointsgaius51 comments

You know, getting these electronics systems ready, sending out to deep space and communicating with these systems is an incredible feat.

But, for me, another amazing thing about these long term projects is the sheer amount of knowledge that needs to be maintained and passed on to new people, over the course of several decades. To pull it off, this need of knowledge management and transfer must be so deeply engraved in the culture of the organization. How do they do it in this day and age of "FAQs", "Forums" and "Helpdesks"? :)

In all seriousness, what do you do to maintain "knowledge" in your organization?

This is a good insight. Having read about NASA's culture some from books like "An Astronaut's Guide to Life on Earth", they tend to be very dedicated to documenting everything. Every last little detail goes into handbooks or manuals about how they fixed problems or things that could have gotten someone killed, adapting procedures to take into account even the smallest of practical details. These engineers come from a different era. You can see all the printed binders on their desks. I think in many ways the process of documentation for space missions is more expensive, pain staking, and solid than modern practices. I think also they do have the advantage of all working in the same place. You just go to Tom, the human search engine/librarian, and ask him what he knows about thing X. (The article mentioned Tom was the self styled "librarian" of the project). Most organizations need a "Tom" of their own and are poorer for not having one.

Our company is building a remote team, and similar to Gitlab, we are discovering that staying in sync is not just a good value but a way of life. Trello, wiki, and code help us repeat our assessment work and ensure everyone is on the same page.

To give some kind of context on more "modern" methods, we use stupendous numbers of PDFs, DOORS for requirements traceability and tons of code commenting. At last count our project had 0.73 commenting lines per line of code (and we use Allman style bracketing).

We are also putting non mission-critical information in a Wiki; basically as a tl;dr of the more dense documentation.

Oh man, I hated DOORS so much. POS software. I dont doubt its useful, but using it was the bane of my existance

Heh, DOORS is pretty crappy. But once you get used to its flavour of crappiness, you can get shit done.

I think it's one of those tools which demand a lot of expertise in setting them up, or you will paint yourself in a corner, with explosive paint.

Everything you described is just standard engineering practice. I've always found it funny how software developers call themselves "engineers" while working nothing like them.

Everything you described is just standard engineering practice. I've always found it funny how software developers call themselves "engineers" while working nothing like them.

So true. But when you have a culture and an industry where 2 years ia a long tenure at a company and a long lifecycle for the latest faddish "framework" or whatever, it's impossible. I'd love to stay somewhere 10+ years and work on a single significant application the whole time. But it seems to be impossible to find such a gig. Experience is generally undervalued throughout our entire industry.

"I'd love to stay somewhere 10+ years and work on a single significant application the whole time. "

Try CAD. The industry is full of crusty applications pushing 30 years. I'm sure some of the careers are equally long.

You can do that at any time. Start a project and work on it forever. I'm still working on a compiler I started in 1982.

It would be standard practice in a waterfall development methodology which big corporations still use to this day.

You document everything and peer review all the time. NASA has an incredible amount of documentation on anything they do. There are controlled processes for everything so whenever you need to go back and see how someone made something, there is an old document for it. The only thing that can't really be passed down is familiarity. You can have the old engineers write down every process, method, or quirk of a system but the new guy still needs to read all of it and get familiar with the system. Long term projects like these involve a lot of learning from other people so that there is a continuous chain of knowledgeable people running the project, like a master and apprentice kind of situation. And if the chain is ever broken, you come back using the documents and information passed on to get up and running as smooth as possible.

I guess beign a goverment agency helps.

After all they're accountable for spending taxpayer dollars on stuff that can crash, burn, explode, and in order to keep funds coming they must be able to prove they're not just making money vanish for no reason.

Mentoring programs, technical data packages (manuals and drawings for everything), a data repository (SharePoint), and a dedicated team of trainers.

I wish this sort of knowledge transfer happened in teams more often.

Even in a relatively small team >20 I have found that even simple communication or sharing of knowledge falls apart _Very_ quickly.

It's something that you need to actively work on, just like automated tests. It also helps if you have someone who has written __good__ documentation before. The kind that people actually reference every few months.

My main thing is that I feel each person/role in a team should _not_ feel Siloed off from one another. That's how I constantly feel - not being included on email chains, meetings, or told thing of importance till the last minute.

Maybe I'm not actually supposed to be in the loop, but it still doesn't feel like how teams are supposed to be ran.

Well, above all, you value experience.

Seriously, the degree to which age discrimination is an accepted reality in tech today is a great failing, and one that has enormous hidden costs over time.

Sadly I know a number of extremely talented older engineers who have been discarded like trash only to be constantly rejected in their search for new employment, until some of them just give up. It's remarkably ignorant and short-sighted of employers who practice this kind of discrimination.

It's not short sighted. Most upper management in those sorts of companies isn't there for any kind of a long haul. They win if they can impress the VCs with what they shipped in the last few months, doesn't matter if none of it works, services are crashing and customers are unhappy. Imagine if organizations like NASA only recognized accomplishments like "added a button to such-and-such dashboard."

It's laziness mixed with incompetence mixed with the simple reality that too many companies consider it a competitive advantage that they're completely fucked up in their process and procedure but manage to somehow churn out something anyway.

And if you're not hitting your targets, why, move the target! Imagine how much easier NASA's job would be if they could say the moon shot was a success even though they only got something into orbit because we redefined what a moon shot was.

> isn't there for any kind of a long haul

That's a 'reasonable' cause for being short sighted, but it still means they are short sighted.

>"And if you're not hitting your targets, why, move the target! Imagine how much easier NASA's job would be if they could say the moon shot was a success even though they only got something into orbit because we redefined what a moon shot was."

This is the way in biomed:

>'When scientists finished the first draft of the human genome, in 2001, and again when they had the final version in 2003, no one lied, exactly. FAQs from the National Institutes of Health refer to the sequence’s “essential completion,” and to the question, “Is the human genome completely sequenced?” they answer, “Yes,” with the caveat — that it’s “as complete as it can be” given available technology.'

https://www.statnews.com/2017/06/20/human-genome-not-fully-s...

I used to work at JPL (and coincidentally was given an opportunity to work on Voyager to update some ground software, but ultimately I passed on it since I was way over committed at the time.)

JPL has a formal database that maintains all its requirements documentation. Though, it isn't very user friendly; and you generally have to know what you are looking for.

When I was there, most of the knowledge transfer happened by talking to the older folks around. As a result, I ended up developing close work relationships with people who had been there for 20-40+ years. The historical context they provided on current and past designs was incredibly useful, and not something you would find in a requirements document.

Software was a different matter. Until recently, when the internal GitHub was introduced, finding and digging through code was not as straightforward. Each team/section maintained their own repos. And there was very little consistency, some used subversion, others cvs, accurev, etc, so it made it difficult to find and read code for projects outside your purview. Naturally, this led to software fiefdoms. In contrast, I work at google now, which is the polar opposite: I can effectively read through the entirety of google3 and send a CL/patch to just about anyone.

>>> How do they do it in this day and age of "FAQs", "Forums" and "Helpdesks"? :)

The secret: Employee retention.

Everything is easier when the knowledge stays in the company.

Easier, yes, but that still doesn't solve the problem over the course of decades; in fact if that is your primary strategy, then you're going to be completely fscked as an organisation when someone _does_ leave (or dies). Employee retention might make things a little quicker to find, but is downright negligent as an institutional knowledge strategy.

There is so much cool stuff in this article. The engineers stuck in the 70s with modern tools to aid them, but that only gets them so far. The love and dedication to this spacecraft. These engineers believe in the mission of space so deeply they married themselves to Voyager. A marriage more substantial than many real marriages. Just thinking about this dedication and how space exploration is, in my opinion, one of the few ways we can truly advance as a species beyond our ruts here on Earth gave me chills. There is something, existential, poetic, and sad about the lonely work these engineers have done for so long.

If you didn't read this article the portraits of the engineers and snapshots of the lives is moving.

I love the Voyagers and I'm very happy that there are people dedicated to keeping us in contact with them as long as they last. These Engineers are splendid examples of taking the long view, something we could use more of these days.

For me the article has a somewhat sad component (next to technological awesomeness):

"In retirement, Zottarelli told me, he would like to see Florida again. He wonders how it has changed. In his garage is a 1954 Swallow Doretti, a fixer-upper. ‘‘It probably needs new brakes,’’ he said. I asked him if there was anywhere he liked to drive for fun. ‘‘No,’’ he replied. ‘‘Not anymore.’’"

Imagine that's your grandfather. With his expertise, shouldn't he be able to 'see Florida twice a year'? I would have wished for a way to have both this awesome technological specialization over long stretches of time, but also given the people on the program some more time to go their own ways. And then the toll of being in the same hierarchy for 30+ years. Ack. But perhaps these people self-selected for stoicism, who am I to judge ;)

That's a really well made point. And for what it's worth, stoicism doesn't ask you to never travel or have fun :)

The most depressing line for me was him effectively waiting (albeit in a joking manner) for his second stroke. No one should be worked to the point of fatalism.

On the one hand I love the team's dedication. On the other hand, I don't think it's just competitive pay that keeps young engineers out of that office. I imagine that the energy and excitement in the early 80s does not really exist anymore, and it's not fair to dump on young engineers for wanting that energy where they work.

The younger engineers ar JPL might rather work on Mars 2020: https://www.jpl.nasa.gov/missions/mars-2020/

One of the proposed instruments is a 1kg helicopter.

They have tried to build a pipeline with the Mars exploration missions, in which senior people can move from mission to mission, and younger people can train up on older, mature missions before they move to newer ones.

This pipeline did not really exist with the Voyagers.

... its oscillator, which allows it to accept a wide range of frequencies, had quit, essentially shrinking the target for transmissions from Earth. Assuming a much narrower bandwidth, and manually subtracting the Doppler effect, they recalibrated their signal. It worked — but to this day, the same calculation must precede every command.

IIRC from an old SciAm article, the temperature of the spacecraft's electronics is also taken into account when doing this calculation.

Yes, the article later says "(On Voyager 2, because of the broken oscillator, any change in temperature also tweaks the receiver frequency.)"

A lot of what has been said here about software documentation and process is true of NASA flight projects, especially manned. Outside of flight, not so much. In some cases, it's a bit more like grad-school-project-that-escaped. An exception would be the modeling code used for things like weather simulation on supercomputers. The scientists need to know that their code actually matches the model they're testing and that things like numerical precision and error bounds are being handled properly. I'll add that the above is anecdotal, based on my experiences at NASA Goddard.

Yup.

I worked in a satellite data office of NASA and there was no formal documentation process. Much of the code running in the facility had been written by a single developer over 20 years, and all of the knowledge left with him. Masterful Perl can be frustrating stuff without documentation or comments.

I recommend seeing The Farthest (https://www.youtube.com/watch?v=znTdk_de_K8) if you have any interest at all in this.

Wow, I got goosebumps from the trailer. Thanks so much for the recommendation.

Looks like it will air on PBS on August 23. http://www.pbs.org/the-farthest/home/

I'd also highly recommend "NASA Voyager 1 & 2 Owners' Workshop Manual" https://smile.amazon.com/gp/product/0857337750

It goes into amazing detail on the history of the spacecraft and the events that made it possible. For example: the story of an intern figuring out the '3 body' problem and, from there, the concept of 'gravity assist' gave me goosebumps

How do you retain such talented people for such a long time? These people could get jobs in the most cutting-edge company in their respective fields. Still they choose to stay at the same place for decades.

By having them work on one of humanity's most important exploration projects ever. I get the impression that a lot of these folks are big picture types and that it would take a lot more than a pay raise to drag them away from their life's work.

It is the love of the mission. The knowledge that you are advancing humanity in this concrete unique way, and that due to circumstance, if you quit it has a tangible increase in the chance of failure for this mission. It is how places like the NSA and other organizations within the government can retain such talented people. It is why PhD scientists toil in their field for much less than they could make in the public sector. It is why people choose to work at a lifestyle company making less than their counterparts at tech giants. A love of the mission/work and or a belief in the organization and its overall goals.

See: Daniel Pink and the whole autonomy, mastery, and purpose idea. This mission checks those three boxes quite strongly, likely captivating the engineers and folks on the mission :)

It's a combination of things. Many (most?) of the people want to be there and are passionate about what they're doing. Financially it's still a great job by normal standards and once you have tenure (I think they may call it something else) it's an extremely secure job.

I have a friend who works there - brilliant guy who went to school for physics but is also very talented at low level embedded systems type development. When he graduated with a physics masters he turned down a six figure offer to start at NASA at around $60k. From what he's told me though, he's essentially guaranteed to hit the six figure mark within 6-8 years. Last time we spoke he had just started his PhD (paid by NASA) and was working on the Mars 2020 rover.

It's also a matter of location.

6 figures in SF/NYC can be less than 60k somewhere else.

And get laid off one year later when the cutting edge is obsoleted.

No thanks.

I guess it depends on one's definition of cutting-edge . . .

> 'The data the probes are collecting are challenging fundamental physics and will provide clues to the biggest of questions: Why did our sun give birth to life only here? Where else, within our solar system or others, are we most likely to find evidence that we are not alone?'

In the end, the engineer's prayer is probably a lot like the astronaut's.

"Please, dear God, don't let me fuck up."

> ...[the heliosphere] blocks 75 percent of cosmic radiation...

It sounds like a solar system-scale Van Allen belt [1] when described like this. So we have at least three layers of cosmic radiation protection identified (heliosphere, outer VAB, inner VAB).

If the heliosphere has implications for high-precision, high-accuracy independent-of-Earth interplanetary spaceflight similar to gravimetric and magnetic readings on Earth do for submarines today, then mapping the heliosphere would be a future asset.

[1] https://en.wikipedia.org/wiki/Van_Allen_radiation_belt

This article is pretty dang cool. Looking over the pictures of the NASA engineers I couldn't help but think "Damn, that engineer looks TIRED". Did anybody else think the same thing?

One thing I've always been amazed at when reading about spacecraft is the incredibly detailed level of control the scientists have over the craft.

It's something I'm not used to here on earth -- when something breaks, I've learned you either crack it open and replace the part or buy a new device :)

Can someone talk (or provide a link) about how this sort of system design works?

I read far too far into the article before I realized it was the mobile site. :)

In the mid 1980's I went on a tour of JPL in Pasadena and actually saw the computers that were (at the time) in charge of recording and storing the telemetry data. I'm vague on the exact details, but apparently the computers were donated from the US Army and were field models (early "portable" computers) and they operated on 48V DC power. So not only were the computers themselves large fridge sized units, they had near matching transformers that were fairly unreliable.

I recall reading that in the mid 90s NASA replaced all the mission control systems for the space shuttle with a single Sun Workstation, so I assume that JPL also at some point replaced the downlink computers for V'ger.

My hat is off to these many fine people who stuck it out with jobs that were probably long periods of drudgery interspersed with moments of sheer terror.

Love the Sun workstation on Enrique Medina's desk 8)

At the tender age of 39, I vividly remember my first IT job, for a bus company of all things, where at age 20 I had a Sparc 5 on my desk. I still love those Sun keyboards.

(We used it to solve for the most efficient driver/ bus scheduling).

I didn't know Sammy Hagar was moon-lighting at NASA. Very impressive.

Reply on news.ycombinator.com