A once-classified anomaly nearly killed NASA's first moon astronauts | Hacker News Reader

A once-classified anomaly nearly killed NASA's first moon astronauts

insider.com

85 pointshsnewman23 comments

It’s amazing how many engineering problems had to be solved in the space program. I heard of one that involves software from programmers that worked for NASA. (I don’t remember the space craft but I think it was Apollo).

After separation from the boosters that got the space craft into orbit, the main engine of the craft had to burn to head off to the moon. This engine could be maneuvered to “steer” the craft, but to propel the craft accurately (without yawing or pitching) the engine had to line up with the craft’s center of mass and the desired velocity vector.

The craft actually calculates the center of mass before the main burn by starting the engine and wiggling the engine a bit then checking the resulting change in the crafts attitude (using gyroscopes, etc.)

The software was designed to do all of this automatically and testing showed that it worked.

However there was a problem. After separation from the booster stages the craft would be in zero gravity so the main engine’s fuel was a big sphere of liquid floating in the center of the fuel tank. The initial burn for the calculation of center of mass could falter if the fuel wasn’t at the base of the tank where the tank connected to the engine.

Before the main engine could be fired the maneuvering jets had to be fired just long enough to press the fuel to the aft side of the tank. The software did this too, but they discovered that in some cases the software didn’t do it long enough.

Fixing the bug was too risky so close to launch, but the software could wildly miscalculate the center of mass and the subsequent burn of the main engine could send the craft into a fatal rapid tumble.

The solution they came up with was to tape a note on the control panel next to the main engine switch telling the astronauts to manually switch on the maneuvering jets to accelerate the craft before ignition of the main engine.

I had a professor that told us about when he found a bug in some of the flight code while he was interning at NASA. They decided it was too late to fix the bug, so they also put a sticky note on the control panel with instructions. Now I'm wonder how many sticky notes were on that control panel.

Sounds like the same bug?

Sounds like the same person, it came up in a class taught by Boyer and Moore.

What you are referring to is an ullage burn. Apollo definitely had ullage burns that mitigated this problem before lighting up the rockets in earnest.

https://en.wikipedia.org/wiki/Ullage_motor

Another note is that it wasn't just for the calculation of how the main burn would go. If the propellant was sufficiently distributed in the tank, not enough propellant would actually get to the valves at the back of the tank, and the engine could potentially not even fire.

The word to google for is "ullage"

There's been a lot of creative solutions. If you have non-cryogenic fluids you can use pressurized bladders. Lots of strange looking (to me) surface tension designs exist to manage the bubble, as long as the bubble is far from the intake who cares. Supposedly the cryogenic Saturn V third stage simply vented its boiloff to the rear such that the stage was essentially under continuous tiny acceleration, which must have made the navigation calculations interesting and set a time limit on how long you could screw around in low orbit before going to the moon.

> Supposedly the cryogenic Saturn V third stage simply vented its boiloff to the rear

This occured after the ullage burn. The vents were not enough to do it alone, but they did keep the propellant settled between the cut off of the ullage burn and the real burn.

https://space.stackexchange.com/questions/31618/why-did-the-...

Also, I believe the Saturn V also had baffled propellant tanks to help keep the propellants from getting too unsettled even in zero-g.

Interesting. Here's a short 3D visualization of the Apollo mission that briefly mentions the floating fuel issue / resolution: https://youtu.be/r0ubiU3PRHw?t=63

Huh.

When playing Kerbal Space Program, I usually design the spacecraft with a final-stage engine + fuel tank below the heat shield, separated by a decoupler. I deorbit the craft and then detach the propulsive part. As my capsule screams through the air in a ball of plasma, I often see the propulsive part following it through the air at a distance of couple hundred meters, almost to the very end.

I always thought this was some KSP aerodynamics bug - intuition told me that the detached part, not being particularly well-shaped, should burn up much quicker and decelerate much faster. Turns out I was wrong - according to the article, this happened multiple times during the Apollo program. I guess this confirms that KSP is not that far from how the Space Race looked like...

I wish the article had gone into why this happened. I also expect the jettisoned engine stages to be much more susceptible to atmospheric drag. Maybe it's because the atmosphere is so thin up there that despite the plasma buildup, it's still basically a vacuum?

The NASA report [1] goes into the details. The procedure was to fire the service module reaction control thrusters to separate it from the command module, spin it to give it stability, and then exhaust their fuel in slowing it down. As the service module was not symmetric around its longitudinal axis, however, the initial spin was not exactly aligned along that axis, and would precess with a period of about 10 seconds. This would lead to sloshing of the residual service module propellants, exacerbating the misalignment markedly -- sometimes wobbling beyond 90 degrees. This meant that the thrusters were not achieving the desired change in velocity, and might sometimes even accelerate towards the command module.

After modeling the problem for various residual fuel masses, a solution was found in reducing the time spent on spinning the service module to two seconds, and cutting off all thrusters after 25 seconds.

[1] https://archive.org/details/nasa_techdoc_19710017109/page/n3

Things of a similar density and frontal area fall thru the air at about the same rate in general. Think of the space shuttle re-entry tragedy where the majority of the parts fell to ground in a sort of shotgun tight pattern, no random parts surprisingly hit Hawaii or glided over to Japan.

Maybe another analogy is they used to make lead shot by dripping liquid lead down a couple hundred foot tall tower and the balls always seem to land about the same place although they're not identical compared to ball bearing production for example.

A lot of the expensive and interesting aerodynamics research into blunt bottomed re-entry vehicles for spacecraft and ICBM payloads goes into the difficult task of getting a stable non spinning stable capsule and a reasonably high lift to drag ratio for moving around to navigate into a target, IF you need to move around, which was needed a lot more for ICBM MIRV warheads than for manned capsules.

Ironically the navigation of the manned programs was excellent such that they didn't need to use much of the possible L/D ratio maneuvering on re-entry, can just reenter the command module ballistically without requiring any navigation corrections, and if the service module is also coming in ballistically because of a software bug, they're gonna end up pretty close together. Now if pre-re-entry navigation was messed up (maybe on purpose to keep the SM and CM apart, as it probably should have been designed?) such that on re-entry the command module had to tilt slightly to use its L/D ratio to glide slightly north perhaps 200 miles, then rather rapidly the CM under controlled flight would be 200 miles away from the uncontrolled ballistic SM. However, navigation was so perfect they both just kinda fell into place...

I found a nice report from googleing from NTRS where they analyzed the data from the actual Apollo 4 test flight and the CM had a ridiculous near straight line L/D ratio between 0.36 and 0.38 for almost the entire re-entry, probably as designed, which must have made the flight computer math requirements very simple for those old days. Of course if you are dead on course you don't need any navigation corrections on re-entry at all, resulting in the SM almost landing on your head because it was uncontrolled due to an oversight.

TL;DR

After being jettisoned, the Service Module remained dangerously close to the Command Module during re-entry, so there was a risk that the two could have collided

That was a stupidly long article, repeating things a lot of people already knew, for that little bit of information...

It’s written in the modern clickbait style- they tell you that you’ll never believe something, then they string you along as long as they can. It’s frustrating to read and usually ends up being mundane info. Usually by the second paragraph you can spot this type of writing and need to be ready for disappointment.

I’m not sure that the whole command/service module separation issue was actually classified, or if it is just a less commonly known part of the story.

I don't know who said it first, but we see a lot of blog posts that could have been condensed into tweets, and a lot of books could be condensed into a blog posts.

Its saving grace, IMHO, is that it has a link to the NASA report on the issue: https://archive.org/details/nasa_techdoc_19710017109/page/n3

Apollo 11 Mission Anomaly Report 3 - Service Module Entry

Conclusion

Tip-off moments applied to the service module at jettison cause the spin vector to be misaligned with the service module X-axis. The rigid body spin motion of the service module excites longitudinal slosh of the propellants in tanks. The sloshing then becomes the dominant force and causes the spin vector to approach a position normal to the service module X-axis. The sloshing can orient the service module spin axis such that the net -X thrusting over a period of 300 seconds can not only reduce the separation velocity of the service module but also reverse its direction. This condition introduces a remote possibility of recontact between the service modules and command module. An optimum separation velocity can be obtained for a range of propellant loads by restricting the spin thrusting to 2 seconds and the X-axis thrusting to 25 seconds of firing time.

I love how the solution was to add one time-delay relay into the circuit. It's a 70s version of a one-line patch.

What makes it the more significant is that they launched 12 with no fix in place.

It doesn't reveal how they kept the astronauts' comments looking out the window secret. Weren't they broadcast live?

It seems like the fact of the SM and its fragments not falling behind must mean something important about hypersonic flight.

> It doesn't reveal how they kept the astronauts' comments looking out the window secret. Weren't they broadcast live?

The plasma buildup around the vehicle interferes with radio communication -- though I'm not sure in exactly which phases of the reentry.

Or maybe the window orientation made that hard to see?

> Weren't they broadcast live?

Nope. Communications were split into multiple channels, all of which were monitored at the PAO, or Public Affairs Office console in mission control by a NASA official. It was deliberately designed so that the public only heard what NASA wanted them to hear.

I think this article is over-hyping the danger. Sure, the risk of the command module and the service module crashing is nonzero but it's so tiny. They start off being pushed away from each other by the separation and they have different aerodynamic properties. It's not surprising that they continued on similar paths initially since the atmosphere is very thin minimizing the aerodynamic differences. To make an analogy, reentry is like tossing two empty beer cans onto a windswept lake. Sure they might have a chance to hit each other early on but the chances of that are small and if their aerodynamic properties are different (i.e one isn't totally empty) they will get further apart as time goes on. The Apollo program had a list a mile long of bigger risks to spend their time mitigating. This article is just hand wringing.

Reply on news.ycombinator.com