NASA’s IMAGE satellite, lost since 2005, is alive | Hacker News Reader

NASA’s IMAGE satellite, lost since 2005, is alive

skyriddles.wordpress.com

826 pointssohkamyung91 comments

The failure analysis is interesting: https://image.gsfc.nasa.gov/publication/document/IMAGE_FRB_F...

Basically, the spacecraft needs a transponder to communicate with Earth. The transponder is attached to a circuit breaker. For various reasons, the circuit breaker can flip off. When this happens, it sends an "I'm off" status bit to the spacecraft's power control computer, which in turn flips the circuit breaker back on.

The problem is that the transponder circuit breaker, if it switches rapidly, turns itself off without setting the "I'm off" status bit. Since the spacecraft computer doesn't know the transponder circuit breaker is off, it never commands it back on. Consequently, the spacecraft can't send or receive commands and becomes useless.

The paper speculates that one way to break out of this state might be to wait for an eclipse that lasts long enough to drain the onboard batteries, causing the whole spacecraft to lose power. When the spacecraft boots up again after exiting the eclipse restores power, the power control computer will unconditionally command the transponder circuit breaker on, restoring connectivity --- basically, waiting for orbital geometry to turn the whole thing off and on again.

There's more. You might think: "Well, just add a bit of code to the main onboard computer to unconditionally reset the circuit breaker every so often. Hacky, but it works, right?"

Nope. If I'm understanding this right, the power display computer (which doesn't appear to be programmable) is designed to ignore transponder power control commands from the main computer. The paper doesn't explain the reasoning for this resign, but I suspect it's probably to guard against software bugs causing exactly the scenario that the circuit breaker hardware bug caused. So in an attempt to guard against a software bug, the designers lost the ability to work around a hardware bug.

Maybe you could program the software to notice that it hasn't gotten commands for a while and deliberately rotate itself into an orientation that causes a power loss and reset? I'm guessing that the solar panels are arranged in such a way that it wouldn't work, and the attitude control fuel consumption might be excessive.

Wonder if the power display computer might be susceptible to Meltdown, Spectre or Rowhammer? Or is it not programmable in the sense that everything is hardwired…

Not susceptible to the first two because these are almost universally large-process in-order CPUs. But generally "not programmable" in the sense of being hard-wired, or just not having a way of doing remote code execution to perform the update. These systems are _really_ simple.

CPUs used in space are usually much simpler and slower than ours – they put enormous amount of resources on error correction due to radiation levels.

I suspect the CPU in this satellite is a BAE Systems RAD6000, commonly used in NASA spacecrafts. It’s extremely expensive and also brittle due to the materials used. Very hard to get due to export regulations. I tried to source one of these CPU’s outside US once.

Wikipedia at least confirms that.

Do you mean brittle in the physical sense, like a potato chip? That's surprising; I assumed that a launch subjects all components to a fair bit of shaking around, not to mention the steday G forces.

If that's the case, it looks like the chip has a surprisingly modern feature set. For example, based on this article[1], the RSC on which the RAD6000 was based had an on-chip memory controller in 1992.

It also had a branch predictor, however, this line is notable:

> Note that the RSC never speculatively executes prefetched instructions.

This means that the branch predictor only did fetch/decode, and so the chip would not have been susceptible to Spectre, as the pipeline would have been flushed and the correct instructions reloaded as soon as the condition evaluated false, before the branch would have been actually executed.

[1]https://web.archive.org/web/20131004213712/http://zmoore.net...

Meanwhile, in February 2000, at the Intel Developer's Forum, Intel demonstrates an upcoming 32-bit processor, code-named Willamette, running at 1.5 GHz.

That was the Pentium 4 not yet in existence so there is no chance of there being an x86 able to be 'Meltdowned' in space. However you are most definitely thinking along the right lines, but I am sure the NASA boffins would have tried all the hacks they could back in 2007. Or maybe the management made sure this was not possible, much like how the Voyager missions were not supposed to go beyond Saturn yet behind the backs on management everyone made sure those spacecraft could get past the edge of the heliosphere.

Why would it matter? It's not like it's running any untrusted code.

I believe the suggestion was that NASA could update the main computer with their own 'untrusted' to exploit the flaw in the power system, side stepping their own protections. But as mentioned below in other comments, these vulns are likely not present on this hardware

That doesn't make sense to me, aren't we talking about two different pieces of hardware talking together? If they could reflash the "display computer" then the problem would be solved if I understand correctly.

Even if it's really two "applications" running on the same CPU I don't see how spectre/meltdown would help since it's all about getting read-only access to "forbidden" memory. Clearly they don't need that (they know full well what's in memory), they want to be able to modify it.

In such a situation RowHammer might be used to modify the state of an other program sharing the same RAM in the right conditions but if there's one piece of hardware I expect to be hardened against spurious bitflips it's a space probe. We're talking about radiation-hardened ECC-protected RAM after all.

"Have you tried turning it off and on again?"

Far too many times.

There are actually products that do this kind of power cycling. I'm a happy user of a ResetPlug that fixes my WiFi problems by rebooting the router when the connection is down.

(I don't mean this as a spammy ad, just in case other people have as terrible Internet access as Chunghwa is giving me)

TBH you probably need a better WiFi router, not some external device that turns it off and on for you... Can you not at least schedule reboots of your current router? That silly plug is $60! You could have bought a mikrotik wifi router for cheaper than that...

The router is provided by the landlord, who already called out the service people from Chunghwa Telecom to look at it.

They turned it off and on again, and it worked (of course). So they refused to replace it.

And I can't schedule reboots, because it usually fails a few minutes after I start using it when I return to my house (not at a consistent time every day).

Ah that's a bummer. It's a modem/router-combo then. I've never had good experiences with the ISP provided ones...

You should be able to buy your own wifi router and plug it into your router/modem combo. Ideally, you'd then run the combo in bridge mode, but it should work regardless.

should and can, are not the same thing.

Why in the world would is that something you shouldn't do?

>you probably need a better WiFi router. That's the right answer. Last time I rebooted a router or AP: never. Unless there's a power failure they run years on end without intervention (DD-WRT on a Buffalo AP, DD-WRT on Netgear WNR3500's, and a number of managed HP MSM422 & Ubiquiti APs if you are curious).

I think restarting after a single pass through the program is a fairly common technique in real-time systems - I think the computers on the Lunar Modules worked this way. It means that you start from a known state on each iteration, and a further refinement is to allow only forward branches (so 'if', but no loops).

Yup, the aerospace data acquisition system that I've used works this way - each acquisition cycle finishes by resetting the FPGA and coming up from a blank slate.

I shouldn't admit it, but I have some tricky memory leak in a node.js server for my app - I spent an evening trying to find it, but in the end I just added daily restart of an app to cron, and it works beautifully ever since :).

Shouldn't admit it? It's my most recent blog post!

http://designbymobi.us/turn-it-off-and-on-again/

Would it be possible to script a regularly recurring event to run everything on overdrive so as to put sufficient load on the on board battery to guarantee a low voltage event?

The article's author figures that there was no such reset during the first eclipse following the failure (in 2007), but perhaps the batteries have now degraded to the point where they are drained during an eclipse?

> Maybe you could program the software to notice that it hasn't gotten commands for a while and deliberately rotate itself into an orientation that causes a power loss and reset?

heh, some spacecraft buses are designed to do the opposite. they'll go into a spin during emergencies just to guarantee that their solar panels are hit at least occasionally.

> I'm guessing that the solar panels are arranged in such a way that it wouldn't work,

plenty of spacecraft have solar panel arrangements where it is possible for them to not receive any light.

> and the attitude control fuel consumption might be excessive.

spacecraft can (if designed to include them, which is common AFAIK) control their orientation with reaction wheels, no propellent required. (the reaction wheels will become saturated sooner or later, though. when orbiting bodies with significant magnetic fields they can be desaturated through the use of magnetic torquer bars. otherwise they have to use propellent to do it.)

Right. I was imagining the reaction wheel unloading using the propellant. But I don't know anything about this particular design, so the concern may or may not apply.

Simply turning around won’t put any long-term load on the reaction wheels though - you spin up to start reorient the satellite and then spin back down once you’re in the desired orientation. It’d be slow - reaction wheels aren’t really designed for gross maneuvers like this - but this is hardly an urgent situation.

It's the spin up and spin down that actually make the orientation changes, though. You can't spin up infinitely, so once a reaction wheel is spinning as fast as it can, you can't accelerate it further, and you can't rotate the spacecraft in the opposite direction on that axis any longer.

(I don't think reaction wheels are capable of gross orientation changes at all -- only very minor alignment corrections.)

> It's the spin up and spin down that actually make the orientation changes, though. You can't spin up infinitely, so once a reaction wheel is spinning as fast as it can, you can't accelerate it further, and you can't rotate the spacecraft in the opposite direction on that axis any longer.

precisely. angular momentum.

> I don't think reaction wheels are capable of gross orientation changes at all -- only very minor alignment corrections.

nah. as an example, SORCE has nothing but wheels and a torquer bar (or two?), and aims all over the sky.

> basically, waiting for orbital geometry to turn the whole thing off and on again.

NASA calls Dell Support, too?

This isn't the first time amateur astronomers are helping NASA: in 2014 a group of amateur astronomers rebooted a long-lost satellite from the 1970s (ISEE-3). They even managed to fire the thrusters for the first time since 1987 but sadly, contact was lost again later that year:

https://en.wikipedia.org/wiki/International_Cometary_Explore...

They refer to this in the article.

Wow, I didn't know there was such a cool hobby out there waiting for me. I know nothing about modern radio scanning but I'm suddenly fascinated to learn.

I agree, I only learned it recently but there are a few very motivated individuals that track satellites and share a lot of the information. I learned about it during the whole Zuma fiasco, from the SatTrackCam blog [0]. I would definitely recommend reading it if this sort of thing interests you, it awesome to see how these guys do the tracking and the sheer amount of information they infer. The Zuma ones are great, along with the ones about the ISS and the satellite got close to it.

[0] https://sattrackcam.blogspot.com/

Any TLDR for the "Zuma fiasco"?

Edit: I read the wiki, very interesting

> A number of articles published by the amateur satellite tracking community state that if the satellite is still in orbit and operating covertly, they will attempt to locate it visually

https://www.wikiwand.com/en/Zuma_(satellite)

Link that works without JavaScript: https://en.wikipedia.org/wiki/Zuma_%28satellite%29

thank you, i really appreciate

And I got downvoted into oblivion for suggesting just that scenario when the launch "failed."

Also see http://goughlui.com/2016/08/05/diy-build-a-70cm-band-yagi-fo...

https://satnogs.org/

Have a look at amateur radio and satellite operations as a whole. You can, with some persistence and brains have a chat with someone a vast distance away via satellite repeater with nothing but a cheap handheld radio.

I’ve only managed to receive so far due to a cruddy antenna but that was with a $30 Baofeng radio from Amazon.

Note: you need a license but it’s pretty easy to get, at least in the UK and you learn a lot while getting it. It also tends to cost you a lot of money in the end because it’s really interesting :)

Good suggestion. I was thinking the same thing. A cheap radio, eggbeater antenna, and gpredict will get you started. No need for a license just to see if you can hear one, but once you do, you'll probably want to get one. Luckily satellites tend to operate in VHF and UHF, so the lowest license class is all you need.

Yep you don’t need a lot of watts either. You can get a long way on 5W.

A HAM license is easy to get in the US. There are three main classes, and I would expect most people on HN could get the Technician license with little to no studying. Won't provide you world wide communications, but it is still good to have. Most places charge $10 and are done by the local HAM group. Here's a practice test for the levels if anyone is interested. [1] The questions are from a pool, so worst case you could just memorize them.

[1] http://aa9pw.com/radio/

It's the same in the UK with RSGB. Very easy.

Yep, read a couple books I got off the internets, mostly memorized the pool questions and got an expert license I never really used (probably doesn't help I'm not the most social creature) so didn't renew last year. Really easy to get a license since they got rid of the Morse code requirement -- just need to learn a few formulas really.

--edit-- Not in one sitting, IIRC went from tech to expert on the second trip.

Data modes and CW are there for us unsociable individuals :)

https://amzn.com/B011HVUEME $26.95

http://www.pe0sat.vgnet.nl/decoding/unix-tlm-decoding (2015)

You aren't kidding, but I don't have anything like the background to start. Looks like I'm stuck reading about others' exploits. :-/

Edit: Storytime. Several weeks ago, about 2:00am local time, I was letting the dog out into the back yard and saw a very bright object (somewhat brighter than the stars around it) passing from NW to SE, almost overhead. It had a bright-dim-bright frequency of about 10-15 seconds. A satellite is the only option that comes to mind, but I checked several "what's overhead now" websites with no luck.

This isn't what you saw (which would've been too early), but check out this satellite from NZ's first orbital launch: http://www.thehumanitystar.com/

That is very cool. I'll be looking for it in 39 days apparently.

It was probably an Iridium satellite.

https://www.youtube.com/watch?v=MTGVuPr9Epg

https://en.wikipedia.org/wiki/Satellite_flare

I'll bet you are right. That video is exactly what it looked like.

Back in the 80's, people with those now giant satellite TV dishes could point them in the right direction and ever-so-slowly decode pictures coming back from one of NASA's probes with their C-64's or Apple ]['s. I forget which one, but I read a magazine article about it and decided not to try it because dishes were expensive back then.

I remember visiting my dads radio enthusiast friend when I was a kid in the early '90s.

He had a giant dish in the back yard hooked up to a computer with a monochrome monitor. I think he got weather data from it.

Hooray for gifted amateurs. This post made me very happy. I especially liked the scientist who replied to the article saying they have experiment on the sat, and would love to see it wake up.

Here's a good article about these guys:

"Meet the amateur astronomers who track secretive spy satellites for fun"

https://www.popsci.com/zuma-spy-satellite-amateur-astronomer

That actually does sound like fun... but I don't personally have the resources to get into a hobby like that :(

It takes $50 in parts: half for a radio (new Baofeng, used Yaesu or Alinco), $10 for antenna parts for a “tape measure Yagi,” and $5 for a cable. The last $10 can buy some hot coffee for long nights of cold.

An additional $10 will get you the license to transmit and be able to talk to your _local_ HAM group.

Also a good feeling during the read and after. Always nice to read serious hobbyist work with a lot of science. I've read additionally an article mentioned in the comments regarding amateur spy satellites detection. Didn't happen until now that I came across this topic and was kind of fascinated about how it works and the people behind it.

Wow...according to wikipedia(so who knows in this case), NASA is trying to communicate with IMAGE via the DSN 4 days after this blog post.

If it is the case that no one remembered about this satellite, then this really is an amazing find.

Yep. I wish I'd caught the communication attempt on DSN Now: https://eyes.nasa.gov/dsn/dsn.html

I have a sudden urge to somehow make DSN Now my desktop background.

According[1] to NASA's Richard J. Burley, as of just after midnight UTC Jan 25 2018, they haven't yet attempted communication using DSN, but they're working on it:

> I have attached the TLE I got this morning from JSPoC/CARA. Looks like a close match. We are in the process of engaging the Deep Space Network to see if they can get a signal lock. They (JPL/DSN) are in the process of digging up 13-year old configuration files for that attempt. The DSN has evolved since then so some adjustments to their system setups will be required.

> I have no schedule estimate yet for when that first attempt will be made. If we are able to get signal lock and verify that it is IMAGE, we will setup a MINIMAL ops setup to verify telemetry/command and make an initial assessment of the health and viability of the spacecraft bus. I don't have a credible schedule prediction yet for that step either.

> Thank you for all of your efforts for IMAGE. IMAGE made ~39 new discoveries about the Earth's magnetosphere and plasma-sphere. At the time it stopped radiating, NASA HQ ranked IMAGE as the 2nd most valuable space-physics mission flying. A follow-on mission called MMS, with a 4-spacecraft constellation is currently flying, in part, to follow up on these discoveries.

[1]: http://www.satobs.org/seesat/Jan-2018/0192.html

Follow up!

> I just received official confirmation from NASA that IMAGE is indeed alive! See Below!

> Engineers at GSFC have acquired the suspect S-band source using the 4m CTA (Compatibility Test Antenna) here at GSFC (.jpg attached and no I’m not in the picture). They acquired the signal while the target was on ascent at about 2RE. Center frequency (CF) was between 2272.478 and 2273.418. The difference between IMAGE documented CF of 2272.5Mhz can be attributed to expected Doppler. Subcarriers are visible as well 1.7Mhz from CF as expected. The signal strength was oscillating. Plots will be forthcoming. The oscillation is not unexpected given IMAGE’s loss of spin balance.

> All indications so far suggest that this is, in fact, IMAGE.

http://www.satobs.org/seesat/Jan-2018/0203.html

> I have a sudden urge to somehow make DSN Now my desktop background.

Not sure your OS, but if you're a Mac user you can do this pretty easily with GeekTool. I used it for some years as a quasi dashboard on desktop setup and it was pretty nice.

https://www.tynsoe.org/v2/geektool/

Just seeing that there's an antenna always pointed at Voyager 1, and that it's 21 BILLION miles away...that's almost 1 lightday...and it's still working. That obliterates my mind.

IMAGE: https://image.gsfc.nasa.gov/

What are the technical and geographical requirements to do two-way communitcation with low-orbiting satelites? A directional antenna, an SDR-card and a decent amplifier i assume, but what about geography? Would i be able to pull this off in a European capital with surrounding electromagnetic/radio-wave pollution one can expect in a capital?

To transmit you’d also need an amateur radio license, although the requirements usually aren’t that difficult.

Yes, absolutely. The trick is to use a very directional antenna—it listens to what you point it at and blocks out noise from the sides. The canonical tool for this is a handheld Yagi, either home made for a few dollars in tubing and screws or ordered from Arrow Antenna.

What replaces the expensive satellite tracking system? A top notch neural network with high endurance myelin-actuated 6-dof arm. You.

Every so often I wonder what kind of shenanigans I could get up to with the "complementary" satTV dish that came with my apartment if I rigged up some sort of tracking system and messed with the feed head for different frequencies. Those things are everywhere, a real untapped resource.

Official confirmation: http://www.satobs.org/seesat/Jan-2018/0203.html

This is so cool! :-) Not that I would understand the slightest of the technical mumbo-jumbo, but it is fascinating, that hobbyists can do such things, these days.

This is amazing. Great post for HN.

This is just a great post. Very inspiring. And now confirmed by several others, and the original IMAGE teams are all excite.

I haven’t seen PGPlot in ages! I didn’t realize it is still being used!

We (I) use it a lot in the Amateur Radio and Amateur Radio Astronomy community

Thank you for posting this! It was a great read and can't wait to see what it leads to.

Dupe: https://news.ycombinator.com/item?id=16231252

How the hell can I google it after it's abbreviation?

"NASA IMAGE satellite" seems to find the project page for me. IMAGE satellite also finds a lot of news entries regarding this incident that you can probably use to find out more.

It’s cute that amateur astronomers can find this satellite, but aren’t we paying NASA billions of dollars to figure these things out themselves?

https://www.nasa.gov/sites/default/files/atoms/files/fy_2018...

No.

lol, it's cute that we give companies billions upon billions of dollars of ad revenue and don't say anything when they make cheesey useless junk, but we get all upset when NASA spends it to try and have the best and brightest engineers solve useful problems.

This is good news. NASA does not always tell us everything. They keep it a secret stuff like intergalactic drive research, ECT until they can prove it works, so it does not fall into the wrong hands.

> They keep it a secret stuff like intergalactic drive research, ECT until they can prove it works, so it does not fall into the wrong hands.

And who might 'the wrong hands' be?

And how does proving that it works prevent it from falling into the wrong hands?

Reply on news.ycombinator.com