Space Station incident demands independent investigation | Hacker News Reader

Space Station incident demands independent investigation

spectrum.ieee.org

394 pointsmultivac42123 comments

NASA's response, printed as an addendum, contains the sort of non-sequiturs that, contrary to the presumed intention, can only suggest that NASA is doing exactly what is alleged here: hoping a serious issue will go away.

"As shared by NASA's Kathy Lueders and Joel Montalbano in the media telecon following the event, Roscosmos regularly updated NASA and the rest of the international partners on MLM's progress during the approach to station..."

As far as we know, nothing untoward happened during the approach to station. Restricting this comment to this period renders it irrelevant and signals that NASA is trying to avoid the issue.

"When the unexpected thruster firings occurred, flight control teams were able to enact contingency procedures and return the station to normal operations within an hour..."

So there's no need to be concerned over why it happened, how long it took to discover the problem, or the inability of those at NASA who discovered it to counteract it until the ISS was in a position where Roscosmos could intervene? As the article author points out, this "all's well that ends well" attitude is what led to the Challenger and Columbia crashes.

"We would point you to Roscosmos for any specifics on Russian systems/performance/procedures."

So, NASA seems to be saying, it is not our problem - but, of course, anything that threatens the integrity of the ISS most definitely is.

That covers the whole of NASA's response so far to IEEE Spectrum, other than the utterly anodyne "We continue to have confidence in our partnership with Roscosmos to operate the International Space Station."

What I would like to know is why public relations spokespersons think it is in the interests of their organizations to make those organizations look clueless, while simultaneously insulting our intelligence by implying we will fall for this nonsense?

Alternatively, NASA doesn't want to sour the relation with Roscosmos while the investigation is ongoing, and what's said publicly isn't an indication of what happens behind the screens. At this level, there's a lot of politics involved as well.

Or they could simply say we are continuing to investigate. So does sound more like they are trying to get people to ignore it.

You're right, they could be trying to get people to ignore. I think it's more likely they're in an awkward political/diplomatic holding pattern.

They definitely can't say that they are continuing to investigate unless they've been cleared to do so, since investigating starts implying patterns of responsibility.

It's definitely a gong-show.

> That covers the whole of NASA's response so far to IEEE Spectrum

IEEE Spectrum was the same paper that got duped into publishing a story about a self driving car that didn't exist. It's a tabloid piggybacking off of the IEEE's brand recognition. The fact that a NASA spokesperson bothered to reply at all surprises me.

Here is a piece of commentary from a Russian insider covering the launch of Nauka, but not the docking, in Russian [1]. There was a litany of smaller issues with the launch, probably stemming from the fact that the module has been tossed around for 25+ years.

I generally don't expect Roskosmos to make any comments regarding Nauka unless it blows up or something. Part of the reason, apparently, is that Roskosmos is being assessed for new funding (from the blog post, Google translate edited for clarity):

"""

Docking is scheduled for July 29. The future of Russian manned astronautics depends on its success. With Nauka, the Russian segment will be able to operate for another 10 years, and be more effective as a research platform.

The future of the independent Russian Orbital Service Station (ROSS) is also indirectly related to the success of Nauka. With Nauka, Roskosmos gets a chance to show that it has the ability to create and launch manned space stations. This, in turn, increases the likelihood of obtaining the funding for ROSS.

Probably, this is the reason that Roscosmos has been reluctant to voice any of the problems with Nauka. When half a trillion rubles is at stake, you will inevitably lose your voice.

"""

Half a trillion rubles is about $7bn American. I've been hearing about plans for this new station for a few years now. I don't really know if it's real. Involvement with the ISS, where Americans shouldered 80% of the financial burden, is really what kept the Russian space program relevant. So I don't expect us to pull out of the ISS voluntarily. If the ISS ends up being decommissioned this decade, hopefully we can piggy back with the Chinese on their new projects or something. That's assuming we have anything they want, which is a big question mark come year 2030.

[1] https://zelenyikot.livejournal.com/158629.html

Wasn't there a plan for "dual" use, aka the next space station can be equipped with a drive and used as a long range spaceship if needs be?

This sounds like something from a sci-fi novel.

The only tangible new thing in production is the Angara heavy rocket that will replace Protons. It's an actual thing that exists and flies. There is also a new piloted ship (Eagle/Federation) in development tailored for Angara, but I don't expect to see it in serial production until late 2020s.

Beyond these two things, everything else is basically concept art.

FWIW, it actually is part of an old sci-fi novel, featuring in Arthur C. Clarke's 2010¹. China suddenly joins a race to Jupiter by firing up its "space station".

¹ https://en.m.wikipedia.org/wiki/2010:_Odyssey_Two

There were plans to launch a VASIMIR Ion drive for testing - The ISS was the only thing in space with enough electrical power to test it.

Apparently NASA lost interest around 2015.

I think I read something about assembling a manned mars ship out of re-used ISS or new ISS-like parts, but I can't recall where, or if the proposal was serious.

It’d need additional radiation shielding to operate for long periods outside Earth’s magnetosphere. The Lunar Gateway modules are a reasonable blueprint for this.

Also, even with the ISS’s power budget, the VASIMR module wouldn’t operate continuously. The plan was to charge it and then run it for short 15 minute periods. It could operate for longer if it took more power from the ISS. It seems they are considering other options for a flight demonstration.

> hopefully we can piggy back with the Chinese on their new projects or something

The US won't touch China with a ten feet pole for anything space-related (and vice versa). The best the US may see is a bit of cooperation with the EU - assuming we can solve our internal post-covid and euro-skeptic squabbles - and that's it.

He's obviously talking from a Russian point of view.

And thank god. My theory

China and Russia should form their own platform for future space explorations.

The EU should really have their own thing and collab with both West and East. The US can do whatever.

The China/Russia alliance could probably work with "other" BRICS nations not entirely known for space research.

In principle I agree, but there's a lot riding against a proper, non-nasa investigation.

1. No one died, unlike Columbia, there won't be a Congressional committee put together on this.

2. Even if there was a congress committee/investigation done, what changes can actually be instituted? Block any more Russian modules?

That's requiring putting pressure on a foreign country to make changes where the relationship runs a knife edge balancing act of frenemy.

The Russian space program is a shell of its former self since ~2010.

The most realistic course of action (at this moment) of any corrective recommendation is to remove them from the ISS. They've been threatening pulling out for a few years now, so let them?

We've already called out the Russians and pressured them for sending up anti-satellite weaponry in 2017 and 2020. Why trust them here.

Another issue - they've been doing things like having all their folks shelter in their modules when SpaceX docks - but not doing that for this type of stuff.

Also the somewhat odd cracks at SpaceX

Rogozin told Russian media that he doesn’t believe SpaceX can build better rocket engines than Russia can. “Musk is not a technical expert in this matter,” Rogozin said. “He just doesn’t understand what this is about.”

Will be interesting to see how Raptor 2 competes (and yes, Russia has had amazing engines).

SpaceX destroyed the main funding for their space agency.

Raptor 1 is already better then any Russian engine along most metrics.

And Rogozin is a journalist made oligarch, he doesn't know shit. Accusing Elon of not being an expert is quite funny coming from him.

I don't think he was speaking to an international audience when he said that, even though it may appear to be like that. Russia is internally quite fragile and to admit that they lost their edge is something that comes with a pretty high price so - just like it always was - they engage in weak PR.

I am curious as to how you would remove us from the ISS. A third of the pressurized modules on the ISS are Russian [1]. Do we just take them with us when we leave? How do we achieve that?

[1] https://en.wikipedia.org/wiki/Assembly_of_the_International_...

"""

The ISS is made up of 16 pressurized modules: five Russian modules ( Zarya, Pirs, Zvezda, Poisk and Rassvet), eight US modules (BEAM, Leonardo, Harmony, Quest, Tranquility, Unity, Cupola, and Destiny), two Japanese modules (the JEM-ELM-PS and JEM-PM) and one European module (Columbus).

"""

*Pirs was undocked to make place for Nauka.

Probably US would pay Russia for keeping them in place.

Russia wouldn’t let the US operate them.

If we imagine an offer to buy out 1/3 of a space station, how much would it be? The amount should be colossal.

It would not be any more expensive than what Russia spent to build that portion. Moreover, accounting for depreciation, it’s not at all colossal.

If it were reasonably easy to just replace the existing modules, they coukd have been replaced.

But if you want these specific modules, already integrated into the station, which are an opposite of a commodity, the price includes a significant premium.

I suppose in theory the US could just starve Roscosmos of funding by cutting back on implicit subsidies. That would require having a contingency in place to keep the station viable without Russian involvement. The problem is that would be massively expensive. As you say, that’s a third of the station. In practice you’re quite right, that plan is completely unviable.

Well it's doable, the problem is that it would kill the ISS as a station. If other parties are ok with it then sure, but I somehow doubt they are.

Removing the Russian segment from the ISS isn't feasible, as it has the only module capable of GNC, and Progress is the only currently flying craft that can raise the ISS's orbit (which, being in LEO, slowly decays).

Russians also can't remove Zarya anyway because they don't own it. The US purchased the module decades ago when Russians had no money. (They still have no money, but they used to have no money, too.)

> No one died

In areas with a strong and proactive safety culture, "near miss" incidents are investigated and mitigated. These "near misses" are cases where "no one died" - the accident did not happen, but if one more thing had gone wrong, it would have.

The manta is "for every _x_ near misses, there will be an actual accident". Where x could be e.g. 10; and accident in this case could mean "total loss of the ISS and everyone on board"

This is IMHO a strong argument for a thorough investigation, in some form. Congressional committee or not.

https://en.wikipedia.org/wiki/Near_miss_(safety)

https://etraintoday.com/blog/near-miss-vs-an-accident/

Clearly there will be an internal investigation with NASA (too much happened outside of protocol) but we'll never hear the full details nor know of the corrective recommendations or lack there of.

If you want to guarantee public knowledge it's got to be Congressional and I just don't see that being realistic because again... No one died.

> but we'll never hear the full detail

This is a pity, good safety cultures are also more open.

There should be an internal investigation at NASA, but they’ve dropped the ball on that several times in living memory. It’s a question of how much has changed culturally since Columbia.

What happened in 2010? Is that when the Falcon 9 started launching?

> 2. Even if there was a congress committee/investigation done, what changes can actually be instituted? Block any more Russian modules?

There are certainly less dramatic measures that could have prevented this incident.

To quote the article:

> the Nauka module's autopilot apparently decided it was supposed to fly away from the station.

Having such decisions protected by a physical switch comes to mind.

A joint committee by all the nations involved in the ISS could require such measures before any module or spacecraft is allowed to dock.

Establishing emergency overrides for firing thrusters to all ground control stations could have greatly reduced the impact.

Why was the unexpected firing of thrusters detected so late? Better telemetry and their analysis could help.

And so on. Lots of things that could be done if there was the will do to do them.

Many more things could be done in response to a through root-cause analysis.

> The most realistic course of action (at this moment) of any corrective recommendation is to remove them from the ISS. They've been threatening pulling out for a few years now, so let them?

From my experience of life in Russia, I'd say this is one of the most effective ways dealing with people habitually resorting to the "do ... or we will sink together!" blackmail.

Let the saboteur succeed, and exceed.

You can swim, and the blackmailer likely cannot (and choosing the threat exactly for them fearing such outcome.)

If somebody brings a stick of a dynamite on the ship, shouting, and swinging it around, give them 10 more, and say good luck.

Such demeanour is wholly dependent on the other party's assumptions that they are harmed more than if they did nothing, which is the reverse in reality.

> remove them from the ISS

Space Station Freedom

https://en.wikipedia.org/wiki/Space_Station_Freedom

I don't understand how this incident, involving a Russian spacecraft, necessarily indicates that NASA now has a poor safety culture again. The argument seems to be missing a step.

Agree.

The way the article is written you'd think this was NASA's station. How dare NASA let this happen!

It doesn't belong to the US, it isn't NASA's station, it is the International Space Station. The module that tried to take its football and go home was Russian.

The article is criticizing what it sees as NASA's desire to go along with the Russians and just ignore it without a real investigation.

The modules was latched to an American module.

When your left arm decides to "go home" and your right arm doesn't, it's a problem for your right arm too.

If all the US side has are flywheels to counter orientation issues, rather than thrusters, maybe costs are being cut where they should not?

Just as no one carries a winch to counter their vehicle sudden backward acceleration. We just have turning wheels for normal day to day control.

It’s much more efficient just having a sane procedure to check which gear you are engaging.

We have two kinds of brakes.

Actually, most cars have one type of brake, with two modes of activation.

It may be just down to the geometry of the station if the Russian sections are in the most efficient positions to mount the thrusters. Or maybe it’s because you need to have a single control system in control of each system, one for the thrusters and one for the flywheels, and it’s not viable to have those span the US and RU sections.

In practice you need both systems. You can’t just run up flywheels indefinitely, every now and then you need to use thrusters so you can spin them down. Meanwhile flywheels give you fine control that’s difficult to achieve with thrusters, especially on something as big and complex as the station.

Small quibble: the ISS gyroscopes are constant-speed; they apply torque to the station by pivoting their axes. The argument stands, though; if there's a constant need for torque then eventually the gyroscopes will end up all pointing in the same direction (a "singularity") and unable to apply further torque; thrusters will then be needed to balance the gyroscopes being returned to effective orientations.

Ref: https://en.wikipedia.org/wiki/Control_moment_gyroscope

I found this very interesting, thank you for sharing!

The ISS has always been designed as an integrated station of the Russian and US segment. There was never any need to duplicate functionality between both segments, as independent operation of each segment is non-goal.

The core of the argument is in the response to the incident:

> To calm things down, official NASA spokesmen provided very preliminary underestimates in how big and how fast the station's spin had been. These were presented without any caveat that the numbers were unverified—and the real figures turned out to be much worse. The Russian side, for its part, dismissed the attitude deviation as a routine bump in a normal process of automatic docking and proclaimed there would be no formal incident investigation, especially any that would involve their American partners. Indeed, both sides seemed to agree that the sooner the incident was forgotten, the better.

The author, who worked in Mission Control operations in the 1980s, argues that similar signs of degrading safety standards and minimizing mistakes led up to the Challenger tragedy.

Degrading safety is a very plausible explanation. Another one is that they know exactly what went wrong and they cover for each other because the mistake is so ridiculous that it would tarnish their reputation.

That's the same explanation, as thoroughly explained in the article -- politics is degrading and overriding safety.

I disagree - if they are covering for a minor, understood and embarrassing mistake, then it does not mean they haven't learned lessons or that they aren't improving those standards and changing processes internally. It simply means they don't want to talk about it to the public, and want to keep it off the 6pm news.

I'm not saying they *have* learned any lesson or that there isn't incompetence/laziness somewhere. I'm just saying that trying to keep the incident low-key is not itself an indication of that.

As someone who has worked in IT operations, there are plenty of times where I have seen a near-outage occur because of a bad process. And many times, the offending team would quietly get their shit together but also not publicize their mistake, so as not to incur the wrath of execs.

"the module began trying to line itself up in preparation to fire its main engines using an attitude adjustment thruster."

One thing that I am curious about is what would have happened if the module had achieved the alignment it was seeking, either as as result of its own efforts or during the procedure to stop and correct for its actions. If the main engine was capable of firing, would there still be an ISS?

I think it's an alignment relative to the ISS (with the visual markers), so there was really no hope of reaching it.

Yeah... Maybe also figure out who drilled that hole!

Regardless of who did it, that it happened at all is very worrying for the suitability of humans for long term living in space habitats.

It is worrying for the state of the Russian aerospace industry. This has become a pattern.

The state of Russia's space program on the ground is not good. It's kind of shocking that they've managed to maintain the reliability of their Soyuz and ISS-related operations to date, more or less, but it was always just a matter of time until those things started unraveling too.

It's probably the same guy who was trying to mount sensors upside-down, and they didn't fit, so he had to take out an old trusty drill

well his job was to mount the damn sensors

You are not wrong. :)

This is one video of a launch failure which was eventually traced back to an upside down installed attitude sensor: https://youtu.be/ycRVAcZC5R4

At 27 seconds into that video you already know it won't end well.

What hole ?

Maybe this one? https://www.space.com/41738-soyuz-spacecraft-air-leak-hole-o...

If I remember correctly wasn’t it an employee covering up a mistake, that to fully replace the damaged panel would’ve required disassembling/ rebuilding a large part of it from scratch?

Edit: Ah, my mistake. Apparently some years prior that happened, but was caught before the Soyuz capsule launched. If the person responsible for that incident got disappeared, I could imagine whoever made that hole wanted to cover up any mistake

I have long argued that dropping the Russians from this program is the only way to go.

The Russians have turned into a hindrance. Every error that happens is a conspiracy. When one of their workers drilled a whole into their spacecraft they publicly accused Americans astronauts to be spies who sabotaged their hardware.

Those who say 'but then they will work with China', ok, go ahead, destroy their space station. Let China finance their broken industry if they really want to.

NASA needs to move on from ISS and move to a privately operated station with the main focus on the moon and Mars.

by independent, do they mean neither American nor Russian?

The investigation would be about NASA's reaction, not the mistake itself. So Americans investigating Americans about a (potential) American problem.

Russia's incidental to the nature of the trigger.

Dupe: https://news.ycombinator.com/item?id=28103641

How is it possible for thrusters to fire without a) human oversight and b) nobody knowing?

"Computers never lie, kid!"

Simple: the module was out of comm reach at the moment.

That's not 'simple'. The module should be integrated with the other modules such that any control terminal on the station should highlight that information.

Moreover, one would imagine that applying thrust should generally require some oversight and that it should not be automated unless there's an emergency in which case there would be appropriate alarms.

The module will be integrated over the next few months. It's not as simple as just docking the module and expecting all the wiring and software config to kick in automatically.

It could be thou.

It's an autopilot with no communication connection. What would be the alternative, drifting uncontrolled whenever it's outside of communication range?

It's not out of 'communication range' with personnel aboard the station.

An ultra basic integration of alarms and communications would be the minimum requirement to allow personnel to monitor and modify mission critical systems.

It was prior to docking. It was a separate entity which had to fly itself without continuous communication to the station in order to be integrated. The entire problem was that the module did not recognize that it was docked with the station and was trying to maneuver itself. Why would you put in a "tell the station to which you are attached that you are firing your thrusters which should never be fired while attached to the station" signal?

It's not a crazy suggestion to have ability for an emergency stop or docking success signal sent from the station to the module.

The problem was that the "docking success signal" was not properly recieved.

The very presence of an emergency stop system should be a signal that the thrusters should not activate, if you are ever in a situation where you would desire to send an emergency stop signal, it is because that emergency stop system has already failed.

"The problem was that the "docking success signal" was not properly recieved."

Then it's not docked and docking procedures should be observed by all personnel as a serious ongoing operational concern, right?

So, if there is integration, which there hopefully would be, that there wasn't proper integration would imply something bad.

Otherwise, while the crew obviously believed it to be docked, why wasn't the auto-correcting features etc. on the Russian module disabled while it's docked?

Again, as part of the docking procedure?

But they thought it was properly received.

This was a computer bug. Function updateDockingState() executed and returned exactly what it was supposed to, but it never properly updated the value of variable DOCKED, so function checkIfDocked() later returned False when it should have returned true. The exact nature of the bug and how it went unnoticed is yet to be determined, but the issue is not with the theory of operation. No integration is going to fix that. Are you proposing that a manual review of every line of code should be part of the docking procedure?

It is amazingly presumptuous to assume you know how to make a safe spacecraft better than a literal space agency.

Docking brackets can carry electrical contacts and SPI with redundant conductors would more then cover this usecase.

That system exists, the problem is that's the thing that failed here. The module didn't recognize it was attached to the station explicitly because there was an issue with those connections.

I tell you how it is possible: it is both a bug, and a design issue.

To say anything deeper about the whys one would need that investigation.

a) software

b) outer space

From the article: "Meanwhile, the station's automated attitude control system had also noted […]" A spelling mistake that immediately brings to mind The Hitchhiker's Guide to the Galaxy and the Heart of Gold :-)

Edit: it seems that it is not a spelling mistake. "Attitude" has only one meaning to me, but "attitude control" seems to be a technical term that I've never heard of.

Where's the mistake? Attitude is the correct term.

“attitude” is the technical term in aerospace for the orientation of a vehicle. https://aviation.stackexchange.com/questions/35933/what-is-t...

The instrument which shows how you are oriented is the attitude indicator: https://en.m.wikipedia.org/wiki/Attitude_indicator

The piloting skill of turning an upset airplane to right side up is called “unusual attitude recovery”: https://www.cfinotebook.net/notebook/maneuvers-and-procedure...

It is quite an interesting topic. Whole books were written about it. “Fundamentals of Spacecraft Attitude Determination and Control” is one I can warmly recommend. But if you just want a one sentence summary: “use quaternions” seems to be the prevailing wisdom.

Quaternions are a mathematical model. Are they also a mechanical solution?

One can argue that the solution where they add a 4th gimbal to the stable platform to avoid the possibility of a gimbal lock is kinda-sorta a mechanical equivalent for a quaternion. See for example the famous exchange between CapCom and Apollo 11 Command Module Pilot Michael Collins: https://space.stackexchange.com/questions/30953/did-michael-...

But when I wrote that really all I meant is that it is advised to use quaternions to represent the attitude information in the software which fills the gap between the sensors and the actuators.

Attitude is a term in geometry:

https://en.wikipedia.org/wiki/Orientation_(geometry)

ADCS is a well-known term, no spelling mistakes: https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-...

>Edit: it seems that it is not a spelling mistake. "Attitude" has only one meaning to me, but "attitude control" seems to be a technical term that I've never heard of.

Anyone not aware of the definition of "attitude" relative[1] to this discussion should shut up.

[1] Pun intended, not that ahofmann would know

Reply on news.ycombinator.com