How NASA Repaired Voyager 1 from 15B Miles Away | Hacker News Reader

How NASA Repaired Voyager 1 from 15B Miles Away

arstechnica.com

155 pointsthm56 comments

It's astonishing they managed to debug and fix this, given how old and how far away Voyager is. I wonder if currently created designs are so robust in terms of repairability. Obviously, everything is much higher-level now; is it possible to fall back to something like low-level, direct memory access communication if something in higher layers fails?

I'm glad to see so many senior engineers on the team; it must be an exciting lifelong journey to work on the mission.

> Obviously, everything is much higher-level now; is it possible to fall back to something like low-level, direct memory access communication if something in higher layers fails?

I have a feeling that code for NASA spacecrafts is not so high-level, for a reason.

From what I understand current flight control systems are still written in relatively low level languages like C++, with extremely careful memory management strategies.

Yes, C++ may even be too far. Writing C in a very regimented way to run on very old processors by today's standards is 'aerospace grade (tm)'.

Let's not be overly pessimistic. Even today's phones have a low-level debug backdoor. I imagine Voyager3 with today's tech would provide much more interesting data, but it's a matter of time, who wants to wait another 50 years

nasa@earth:~$ ping voyager.1

PING voyager.1 (###.###.###.###) 56(84) bytes of data.

64 bytes from voyager.1 (###.###.###.###): icmp_seq=1 ttl=inf time=44.8 hrs

64 bytes from voyager.1 (###.###.###.###): icmp_seq=2 ttl=inf time=45.1 hrs

64 bytes from voyager.1 (###.###.###.###): icmp_seq=3 ttl=inf time=44.9 hrs

64 bytes from voyager.1 (###.###.###.###): icmp_seq=4 ttl=inf time=45.2 hrs

64 bytes from voyager.1 (###.###.###.###): icmp_seq=5 ttl=inf time=45.0 hrs

IPv4 didnt even exist when it was launched. They should have waited for ipv6

Something very wrong is going on with voyager, your terrestrial network or the universe!!! The spread of that ping is 0.4 hrs which I take to be 24 minutes! During that time a radio signal would travel 3 times the distance between the sun and the earth (AU). The only thing I can think of that could possibly be causing that delay is having to wait for a deep space network dish to aim properly.

I totally lack the patience to be able to deal with sending a command and having to wait 2 days to see the result. Everything about this is awe-inspiring for me.

Imagine opening Vim and gradually typing the commands in to edit a file with a 2-day time lag. Only at the end to notice the fateful words at the bottom of the editor: [read-only].

This reads like a great opening to a geek-sci-fi book!

geek-sci-fi-horror book

I wonder if you could teach millions of terminal sessions to an LLM and then have mosh run the inference algorithm to hide the latency (and eventually correct when it gets the real data)..

> Imagine opening Vim

Imagine being able to close Vim.

And yes, I’m hopeless. Nano please.

Don't get into GenAI then.

It sounds like my old company (not NASA) when we submitted tickets to offshore tech support - 2 days response time was the norm, and sometimes we never got a response back at all...

I told my dad about the Voyager fix, and he said that back in the days of punched cards, latency of 2 days was quite normal! He was still impressed at the high stakes of the repair though, with no option to go over and turn it off and on again.

This is basically why I quit cancer research to go into software. It was like trying to debug via the postal service. You do something to your cell cultures, and you only find out 6 weeks later if something went wrong.

And because a million different things can happen to cells over the course of 6 weeks, you don't get a line number, or a stack trace, or very much debugging information at all.

So you quit potentially life-saving research because you couldn't open a terminal session to a cell culture? Why did you even enter that field in the first place? Genuine question.

./configure --fix-cancer

sudo make cell

Because I legitimately thought it would be a field where I could maximize how much of my brain I used. Instead, it turned out to be 99% dishwashing.

I very much loved undergraduate studies. I legitimately love studying what has been done and what has been discovered already. While we often say that we know nothing (it's somewhat true), the amount of human knowledge that already exists, especially in the life sciences, is unfathomable. And that's genuinely challenging and exciting.

But as soon as you start actually doing the research work, it's a lot less exciting, not much happens, and there is not a lot of problem solving (at least not in comparison to software). It's just a slow methodical grind, with pretty much no pivoting opportunities. You just stick to the plan, and you must try the same thing over and over again. I was surprised by that. Your experiences may differ depending on where you worked, on your funding obligations, and on your specific research area.

And after you finish your PhD, you might get lucky and get one of these fancy $ 40k post-doctoral fellow salaries. /s

(The way science works at a large scale is pretty fantastic, but how it feels for an individual is a very different thing.)

This just sounds like academia needs drastic reforms.

In Engineering we have 22 year olds directing technicians to do that kind of menial work.

The fact that the work is so unvaluable yet years were dedicated to it, seems like a failing somewhere. Not everything needs immediate economic value, but to see supposedly smart people not make 6 figures means:

>They arent that smart, smart engineers get picked up by irrelevant industries simply for being smart

>Someone is directing the smart people poorly.

Of course its both, the leadership of academia has failed us. Those who stick with Academia into PhDs and beyond are essentially second class performers who couldn't make it in Industry.

There are of course a lot of things I would like to change or improve, but I don't know if I share your broad pessimism on academia or research.

Science and engineering are very different endeavours, and while both should take lessons from each other, you just can't approach science in the same way that you approach engineering.

How about actually hooking them up to a computer? https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4490468/

My computer course in high school has a one week turnaround time. Send your punched card deck off Thursday of one week, get the printout back the next Thursday.

Two days would have been luxury.

When you say ‘has a one week turnaround’ you mean ‘had’ don’t you?

If not, I’d quite like to try a course on punchcard usage.

I'm in awe that it's maintainable even though it launched almost 50 years ago, while there is software deployed right now that won't be maintainable in the next 5 years.

Because it makes economic sense to apply extremely rigorous standards when writing software for a space probe, but not for a web app for expense reclaims?

Or maybe because we are, literally, talking about rocket scientists?

Perks of direct control over your systems.

When you inherit some random security update to an obscure library and it causes things to break, you were never expected to understand that edge-case.

However, if you spin up your own safety critical embedded system, you should be able to follow the logic completely.

Different costs. One is in the billions of dollars, one is in the hundreds of thousands of dollars.

True, yet I feel the expectation regarding software has shifted from expecting much better quality even 10-15 years ago to things we have now, where nobody is surprised about web apps crashing your browser etc.

> Through their investigation, Voyager's ground team discovered a single chip responsible for storing a portion of the FDS memory stopped working, probably due to either a cosmic ray hit or a failure of aging hardware. This affected some of the computer's software code.

IIRC, the Voyagers have redundant FDS computers, but I believe the non-active ones have been considered bad for decades. Now that they're seeing hardware failures on the active one, I wonder if they're evaluating the problems in the inactive ones to determine if they might actually be in better shape.

Would anyone be able to hack Voyager? Just wondering - if the software is that old, as long as you can get a signal to it anyone should be able to hack it / wipe it right?

Is the biggest barrier the enormous satellite dish you'd need to contact it or do the commands have a auth header with some key you'd need to brute force?

I think the big radio telescope is the main barrier. At this point there's only one in the world with enough transmit power to talk to it, and a handful that can receive the signal.

https://en.wikipedia.org/wiki/Canberra_Deep_Space_Communicat...

(pdf) https://deepspace.jpl.nasa.gov/files/820-100-H.pdf

70 meter antenna, 80 kW X-band transmit power (122 dBW EIRP)

I posted a Reddit link earlier asking the same question [0]

HN thread [1]

[0]: https://old.reddit.com/r/explainlikeimfive/comments/1cbhvwg/...

[1]: https://news.ycombinator.com/item?id=40141836

> Due to its age, Voyager doesn't have any ground simulators

Has anyone attempted to build this? Assuming the documentation is available

The doc is on paper, not properly organized, and the people who wrote it are mostly dead: https://www.wired.com/story/nasa-voyager-one-space-probe-los...

So they test in production, and they see what happens.

It's a really cool job they have (I'm not sarcastic), life-balance at NASA is known to be great ( = low to no pressure ) + since it's publicly funded, no clients, no profit to make, and if politics ask for feedback, you can claim anything and they can't verify it, and you can play with tech that others don't have the chance to see.

  > It's a really cool job they have (I'm not sarcastic), life-balance at NASA is known to be great ( = low to no pressure ) + since it's publicly funded, no clients, no profit to make [...]

I've known some NASA folks and they would not describe it as "low to no pressure". It's highly bureaucratic, constantly under budget pressure, and contractors get much of the cool work.

Don't know about voyager, but I expect that many of those folks have been there decades-- like a closed society. I am sure it's awesome for them, more power to them, but it's certainly not the norm.

I guess it depends on the team, and I tend to believe what you describe as well.

I didn't know about the contractors.

It still seems like a great place to work for, even in terms of social recognition.

It hit me just now that most of the people that designed programmed and launched this satellite are already dead

This sounds like a miscommunication. They probably don't have a simulator for the entire device, but attempting to fix the software without having an emulator of the computer would be downright irresponsible.

Exactly what I was thinking - it's virtually guaranteed they've spent the effort to build a software emulation solution.

[dupe]