Goes-19 weather satellite enters Safe Hold mode | Hacker News Reader

Goes-19 weather satellite enters Safe Hold mode

spaceweather.gov

178 pointsyabones92 comments

Former GOES engineer here. At this point I'd almost be surprised if 19 didn't have something go wrong. We had issues on almost every other satellite. GOES-17 had the loop heat pipe anomaly(Supposedly from someone stepping on it in the cleanroom...), GOES-15 (IIRC) had a micrometeorite strike, and GOES-13 had a fuel tank anomaly right before deorbit.

GOES-16 and GOES-17 are on-orbit spares, so in the extremely unlikely event of a total failure there's at least another spacecraft on-orbit ready to take up station.

That said, I have every faith in the GOES team to get to the bottom of this. They're the best, and I often wish I was back there working with them.

> Supposedly from someone stepping on it in the cleanroom

I would be too embarrassed to return to work if I did that.

How do they track that? Is there a log book where someone has to write to?

"Observed EMP555 step on loop heat pipe. Conducted visual inspection of the affected area; no damage found, pipe remains nominal."

I'm sure they take detailed close out photos and likely video of the entire handling process.

“Measurements taking of the person: foot and shoe weight, total weight including clothing, height and leg length, estimated velocity of travel at time, estimated duration of load”

Since it went into space with the stepped-on part, clearly nobody DID write it up.

They really don't care. If the satellite blows up or fails, they get to build another.

I didn't but that's the case. More likely there was a limited chance to evaluate the issue and something like they had a limited time launch window, no replacement ready and it tested good. "Just put a little butter on it, it's fine".

From now on, every time I see the word "anomaly" I will assume it is an euphemism for "someone stepped on it".

"Anomaly" is one of those words that, in the space industry, can mean anything from "Our payload isn't behaving as well as we're expecting" to "Someone stepped on it" to "A rock hit it"

You can use it to describe literally anything that's off nominal. It's fantastic.

I learned recently that it can also mean "the spacecraft exploded on the launchpad in an enormous fireball."

https://www.abc.net.au/news/2026-05-29/blue-origin-rocket-ex...

That's already known as "rapid unscheduled disassembly" - and happens so-often in spaceflight that it's hardly anomalous at this point.

I do love the term "lithobraking" as well.

I feel like we're all forgetting the most obvious root cause... "aliens"

Or it has failed catastrophically and gone under voluntary rapid disassembly to help engineers to diagnose the problem: https://www.youtube.com/watch?v=EC9icOKGJ94

> GOES-17 had the loop heat pipe anomaly(Supposedly from someone stepping on it in the cleanroom...)

Was this reported anywhere in the news? Sounds like one of those "not even once" kind of mistakes.

You expect cleanrooms to be extremely controlled environments where skilled technicians and engineers very carefully handle sensitive equipment... Then someone steps on a component :D

Yeah. Stuff happens.

There is the famous case of dropping the NOAA N-Prime weather satellite on the ground. They were trying to turn it from vertical to horizontal and they forgot to bolt it to the adapter. Worse: multiple people signed the paperwork attesting that they verified that it was bolted down correctly. Pictures and details here: https://spaceflightnow.com/news/n0410/04noaanreport/

Or the famous “space aligator” event. There the original plan was to launch a manned space craft and an unmanned module to do a docking test between them. But the shroud protecting the target module didn’t deploy properly. If i remember it right because the man who usually assembled that part had to leave during assembly because his wife was giving birth. Someone put it together but appearantely not correctly. The half opened shroud reminded the astronauts to the jaws of an angry aligator hence the name. Pictures: https://www.nasa.gov/missions/gemini/gemini-ix/gemini-ix-cre...

Or to not only list American failures: this russian satelite launch failed spectacularly because the IMU was installed upside down. https://youtu.be/ycRVAcZC5R4?si=LRTS7sutKSp6HnGs This was designed to be impossible to do, but someone bent and forced the component to stay in place in the wrong orientation. Then someone else whose job was to check it just signed the paperwork without climbing into the location where he could have checked it.

I love these cases. Because it shows to me how even though the trappings of high-tech we are all fundamentally just occasionally lazy, occasionally distracted monkeys banging rocks together.

> multiple people signed the paperwork attesting that they verified that it was bolted down correctly.

In my experience, the more people who are assigned to do something, the more those people assume that someone else did it and slack off. Of course, I've never worked on something as difficult to fix as an orbital satellite.

The infamous "it fell in-between the seat and the console" problem.

These happen constantly. It's always important to close the loop on these and make sure someone is assigned to it directly, not left to indirectly assume.

> Stuff happens.

> There is the famous case of dropping the NOAA N-Prime weather satellite on the ground. They were trying to turn it from vertical to horizontal and they forgot to bolt it to the adapter. Worse: multiple people signed the paperwork attesting that they verified that it was bolted down correctly.

That wasn't a case of "stuff happens." It was a case of systemic issues with defense contractor employees not doing the work they were supposed to be, cutting corners, and "not giving a shit because if it breaks the government will just order another one from us."

It was basically fraud/grift/incompetence, not "stuff happens."

> If I remember it right because the man who usually assembled that part had to leave during assembly because his wife was giving birth. Someone put it together but appearantely not correctly.

And why was the process so poorly documented only one guy could do it right?

Why didn't someone check the assembly?

Answer: defense contractor laziness and incompetence.

Upside IMU is my favorite if only for the stupidity and spectacular video that resulted.

There was also a test of the Apollo Launch Escape System wherein an unmanned CSM was launched on a Little Joe rocket with the plan being to manually trigger the LES at a specific point in the flight. The problem was that the roll accelerometer in the Little Joe had been installed backwards, so it quickly developed a substantial roll rate once it left the ground and the resulting centrifugal force acting on the innards of the SM turned them into rapidly escaping outards. This tore the SM skin apart and satisfied the automatic abort conditions for firing the LES. The LES operated as designed, the CM was pulled free of the SM, then went on to become stable and reach successful recovery conditions. The result was determined to be a successful test of the LES and the mission did not need to be reflown.

In 1968, a team of Perkin-Elmer technicians were carrying the million dollar Copernicus space telescope mirror to the other side of the lab. They tripped over a piece of scrap lumber. Shaka, when the walls fell.

That's a new one for me. Do you happen to have any pointers to more info? I found a few mentions, but nothing substantial.

It was mentioned in http://www.terrydunkle.com/glass.php and in a Sky & Telescope magazine article from 1989 which explained why Perkin-Elmer began using very meticulous step by step procedure checklists after that. To prevent any major mistakes. Oh the irony.

NOAA-19 was dropped on its side because somebody forgot to attach the bolts to the transport platform it was riding on.

https://www.google.com/search?client=firefox-b-d&q=satellite...

The independent review team concluded the issue was from FOD in the pipe and recommended a redesign. Do you have a definite citation for someone stepped on it?

From the latest update (https://www.ospo.noaa.gov/operations/goes/status.html#datafi...), it looks like they're already restoring systems.

``` Update #3: DCS and SAR have returned to service as of 1630Z. Engineers will now work to restore ABI and expect imaging to resume by 1900Z. Image navigation may be slightly degraded for the first hour after imaging starts. The GOES-19 instruments will be restored in the following order:

    ABI
    GLM
    SUVI
    CCOR-1/EXIS/MAG/SEISS

The recovery process to return all GOES-19 instruments to normal operations is projected to take approximately 8 hours.

Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known. ```

Looks like its started to publish new imagery again! I'm sure those fellows are relieved to have it working... I can't imagine the stress of trying to debug something that far away and unreachable!

https://www.star.nesdis.noaa.gov/GOES/sector_band.php?sat=G1...

https://www.nola.com/news/hurricane/weather-satellite-goes-1... explains a bit more what this is, and what this means.

> The main NOAA satellite for tracking Atlantic, Gulf Coast hurricanes is out until further notice

> GOES-19 is the main instrument used to identify tropical waves as they strengthen and move over the Atlantic, Caribbean and Gulf of Mexico, providing real-time tracking for forecasting.

I love how "safe mode" for a satellite is basically: "extend solar panels, turn self towards sun, don't do anything unnecessary, wait for further instructions".

They should rebrand it as "Praise the Sun" mode. We are sorry, GOES-19 is temporarily unavailable during a planned solar worship break of indefinite length.

Initiate Sol Invictus mode

The cool thing about geostationary orbits is that they're far enough out that they get 24/7 sun (Except around the equinoxes). We could easily fit a solar worship break in the schedule in between imaging and momentum dumps.

> a solar worship break

It'd be easy to optimize with some additional religions too, since from that distance pretty much all possible Earthly [0] holy sites are in the same direction.

[0] I just stumbled onto a fun grammar question! Is it "earthly" (non-supernatural) or is it Earthly (proper-adjective related to the planet), or both? I submit that it's possible to have an "earthly Martian holy-site", and therefore only the capitalized version works here. :p

Earthian, I believe? If you adhere to the same rules as Martian, Venusian, et al. I’d me more inclined to use Earthen, capitalized, though.

Terran

Sure, GEO sats are almost always in full sunlight. But still, if you loose orientation, you could end up with the solar panel wings pointing in the wrong direction or overheating parts of the satellite not meant to be in full sunlight for long time.

Just don’t point the thing at any maggots.

My restorative mode is also "turn self towards sun, don't do anything unnecessary".

"Turn your face to the sun and the shadows will fall behind you." Māori Proverb

Interestingly, I noticed this in aproximately real time. I had been checking up on the visible-light geocolor composite images every hour or so to look at the massive plume of Canadian wildfire smoke that was turning the skies in the northeast dark orange yesterday.

I haven't interacted with the GOES site or cared too much about the image output until the last 2 days, and the it immediately broke. Somewhat humorous to me.

Is this what you're talking about? https://www.star.nesdis.noaa.gov/goes/conus_band.php?sat=G16...

This is a really cool page, never really thought about how much information you can squeeze out of just IR (recommend selecting the dust RGB and reading the PDF it links to explaining just how dust is tracked etc)

I strongly recommend the Downlink app for this!

https://downlinkapp.com

It downloads false-color images from GOES and Himawari NRT and sets them as desktop images.

Looks like they're making progress toward getting things restarted: "Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known." [0]

[0] https://www.ospo.noaa.gov/data/messages/2026/07/MSG_20260716...

Anyone interested in accessing GOES data at scale will find this interesting - I created a Zarr index over the 7 billion chunks of data in the GOES-16 archive.

https://www.earthmover.io/blog/virtual-zarr

I used to work for NOAA. I think the scientists would flip out over this. Did you share it?

Yes! But if you want to share it with anyone else that would be great, since we're advocating for fairly radical changes within a big bureaucracy here, as I'm sure you will appreciate :)

As an aside, I'm always surprised how US Gov websites look like they've been made in Dreamweaver in about 2006. Not even seemingly with a emphasis on usability either.

The ones that look old are old. The USG has newer design systems that you'll see used on many of the websites that have been redesigned more recently: https://designsystem.digital.gov/

This admin gutted both NOAAs budget and workforce so a website redesign is probably low priority at the moment.

Sites like NASA's APOD have not changed by design. So many third parties have been built up around sites that any change [w|c]ould break so much for no effective gain. Same holds true when people ask why things like NOTAMs and even NOAA's alerts are formatted the way they are.

A new feed could be created and people gradually migrated to it.

While it may not be flashy, I personally find the GOES sites extremely useful. Things are often simply placed at obvious and expected URLs, so scraping or monitoring is extremely easy.

I wrote the script that provides the GOES NavSum [1] and it pretty much just builds a standardized text file and drops it in the folder. The neat thing is that this makes it really easy to programmatically scrape and parse the data.

I wrote a personal script at one point that would download the GOES-EAST CONUS image and both EAST and WEST full disk images and composite them into a wallpaper. At one point my server had 500GB of archived GOES imagery. I liked to joke with my former coworkers that I could report image anomalies before they notice because my desktop wallpaper would change every 10 minutes.

[1] https://www.ospo.noaa.gov/resources/cemscs/navsum.txt

Maybe if the UX was nicer, you wouldn't need to write scrapers and parsers and could just use their site.

They're scraping to automatically update the wallpaper on their desktop. That's not something a website can do, even with fantastic UX.

"That's not something a website can do, even with fantastic UX."

Actually, I used to have a live-updating website AS MY BACKGROUND. Windows 98 and Me, website used AJAX and Comet to make it happen.

You used to be able to set websites as your desktop background.

I'm sure that was 100% secure, too. /s (I did the same thing back then.)

We don't need a bloated React framework to show a plaintext file with the fuel tank levels. It's NOAA, not Microsoft.

> We don't need a bloated React framework

Could have stopped there for 99% of websites

Hey, I have a script for updating my background too! I'm not archiving the old images though, but I've thought about it to make some cool animations

Hah originally making an animation was my plan, but as so often happens it fell on the backburner and then I ended up with a massive archive. I just deleted it once I realized that A) Better archives exist elsewhere and B) I wasn't going to do anything with it.

I still have the script somewhere. I should throw an LLM at it and see if I can't sand off a few rough edges.

It would be great if an LLM could be trained to generate interpolated images between the 10m intervals of the full disk geocolor product. The animations would be fantastic.

I've got about 2.5 years worth of that imagery if someone knows a good way to do this on a budget.

I doubt you need an LLM for that, a diffusion model or perhaps even a deep CNN could probably do a passable job. You could train it by taking consecutive triplets of images [A, B, C] and providing [A, C] as inputs with B as the expected output.

Thanks, that was poorly worded.

make torrent of it

There's an app that updates the desktop https://downlinkapp.com/

Why must everything look and be modern

The counter to this is the UK.Gov website. Very simple and basic, modern (maybe) but very accessible. Government websites should put accessibility over everything else.

They even have their own design system and CSS framework: https://design-system.service.gov.uk/get-started/

You can thank AccuWeather for nerfing any funding for site modernization. I'm surprised the tiled radar map hasn't had the Biden performance fixes reverted.

The link OP submitted appears to be a webpage displaying a screenshot of another web page, and the image aspect ratio has been altered. It's so comically bad it had to be on purpose, or someone is doing their web dev in MS Word.

Edit: I think actually it's a screenshot of a screenshot even, and this appears to be the entire design of spaceweather.gov. What in the holy heck is going on there? This has to be a top 10 worst website designs of all time.

Lots of the web still looks like this when you step outside the comfort zone of big tech search engines, content streaming sites, and social media.

status page says

        SAFEHOLD HAS BEEN RESOLVED

> Update #2: The GOES-19 Safehold has been resolved and engineers are working to prepare for restart of the onboard instruments. More information on the recovery timeline will be provided when known.

* https://www.ospo.noaa.gov/operations/goes/status.html

* https://www.ospo.noaa.gov/operations/goes/status.html#datafi...

A safehold is like maintenance mode, shutting down all non-essential systems, after it detects something is wrong. Doesn't necessarily mean it is gone for good, but not a good sign.

What is it that the space aliens don't want us to see? The obvious conclusion is that they are hiding their invasion fleet arrivals inside of hurricanes. The proof will be when the system comes back online and only permits us to see ordinary weather.

We really need to finish the Gollop Chamber.

They're concerned because GOES-19 deployed with the newest Rothschild Space Weather Laser, which could disrupt their atmospheric camouflage

What would a space hurricane look like?

https://memory-alpha.fandom.com/wiki/Lightning_storm

Very unfortunate timing given the ongoing wildfires and associated smoke spreading across eastern North America in recent days.

These environmental satellites affect our day-to-day including the accuracy of weather forecasts. I believe the GOES program must receive more government funding because GOES-19 is a single point of failure and there have been numerous funding concerns. This is a national security issue because lack of resources to mitigate emergency weather events can lead to increased loss of life.

What about the on-orbit spares?

is this old cache?

* https://cdn.star.nesdis.noaa.gov/GOES19/ABI/CONUS/GEOCOLOR/2...

* https://cdn.star.nesdis.noaa.gov/GOES19/ABI/CONUS/GEOCOLOR/G...

Looking at the timestamp, that's from yesterday. Nominal product delivery happens ~every 10 minutes.

Better link: https://www.ospo.noaa.gov/operations/goes/status.html

I disagree. That just shows GOES-19 as "green", whatever that means. The OP link is also not very informative, but this link is even less so.

The outage list at the top is up to date, but the main status page is nearly three months old - the last updated date at the end is April 20, 2026.

> Please note: This status information on this website is generally updated on a monthly basis. Recent outages and anomalies on data flow are highlighted at the top of the page.

https://www.ospo.noaa.gov/#datafile-3

Invasion of Cuba incoming?

No, you can’t see ships with this satellite. Too small. Besides, Russia and China have way better satellites that they’ll happily share intel from with Cuba.

Reply on news.ycombinator.com