Apollo 11 Had a Hidden Hero: Software | Hacker News Reader

Apollo 11 Had a Hidden Hero: Software

wsj.com

204 pointspsim158 comments

I recently started listening to a fantastic podcast called 13 Minutes To The Moon[0] produced by the BBC. It’s a retelling of the last 13 minutes until the Apollo 11 lander touched down. Episode 2 talked about the software and the 1202 program code. They interviewed the Guidance controller who was only 26 at the time and talked about how the average age of the flight controllers was 27. I thought that was incredible, you don’t see that much responsibility placed in such junior people anymore.

[0] https://www.bbc.co.uk/programmes/w13xttx2

The American and Canadian military are significant examples where many organizations are 'run' by people under 30. Yes, most field-grade and general officers, and senior NCOs are over 35. But, for example, the day-to-day decisions of logistics and tactics and technological is done by people between 19 and 27, by junior enlisted and company-grade officers.

I have never had as much responsibility an operational control over my job, in both fiscal and mission terms, as I had between the ages of 19 and 23 while in the military.

Just a thought - is this the generation that never gave up in power and make decision so younger one never has a chance.

There are a couple of counter examples, particularly in New Space startups. At 26, I was a Mission Director (SpaceX's equivalent to a NASA Flight Director) for Dragon re-entry and landing.

As the old Apollo-generation engineers start to retire in the space industry, there _is_ fresh blood coming in, and even at larger companies, I've found young folks in top positions. It's just not widely advertised.

I wonder how much of the low age is because it was a relatively new organization at the time. As a counter example, NASA's current average age is 50+ in many organizations.

It's a blessing and a curse. Aging organizations retain lots of organizational knowledge but also tend to exhibit a lot of stagnation/complacency.

Yes, the podcast attributed it to NASA being a new organization and there being nobody with experience in most of the jobs. There were aircraft engineers and a small number of rocket engineers before, but NASA needed vast numbers of new employees and for jobs like mission control that just didn't exist before.

Some people were being hired withouth interviews it mentioned too it mentioned were there repercussions to this?

Quite a change from answering all these bloody mediaeval riddles

It's a really good podcast, but I found it a bit bloated in places, I could have done without the Hans Zimmer theme, of which they are so inordinately proud.

> of which they are so inordinately proud

If you got a multi-Grammy winning composer to make your podcast theme you'd be extremely proud too.

That’s funny because I actually love the Hans Zimmer theme.

I really loved the Hans Zimmer music too. I got pumped up every time I heard it. I felt some bloat at times when they really dug into that 1202 error. It just kept coming up again and again.

Yes - it's bloated for a reason ... all of the things they've spent long episodes on are so you know what's going on when they do the realtime recordings of cap com.

"you don’t see that much responsibility placed in such junior people anymore"

Sure you do, typically in new industries, like internet companies today, or space in the 60s.

Eventually the unicorns will be manned by aging suits, and JavaScript will be the new Cobol, and the hip young things will think vowels in names are actually kinda cool.

CuriousMarc's Apollo Guidance Computer (AGC) restoration is a must-watch: https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...

I’ll second that. I particularly liked the part on home the memory worked

I'm amazed at the technical skill and knowledge he and his team has. Fabricating simulations, replacement parts, unpotting modules and repairing them, etc, etc, etc.

I thought the Xerox Alto restoration was amazing - this beats it handily.

There is a good book on the computer systems of Apollo project: Digital Apollo https://mitpress.mit.edu/books/digital-apollo

It describes interesting questions that were faced on the project: say, how much control over a lunar module could be given to a computer system, and how much they should rely on human engagement.

I read this book a couple years ago, it’s great. In fact, I picked up the book 2nd hand with a note from the author inscribed on a cover page. Neat.

This book is especially interesting to anyone building systems where humans are required to interface with a computer on a “mission-critical” level.

It goes in-depth about the new type of electrical component-integrated circuits used in the guidance system. And more about other aspects of the navigation systems used on Apollo.

I recommend.

It's interesting to me that 50+ years later this debate on system control is still very much relevant to aerospace today, like the recent 737-MAX issue.

There's not really any debate. The MAX ignored very basic controls lessons that were learned and institutionalized, as you say, over 50 years ago.

I agree that the 737-MAX wasn't a good example.

My limited understanding in a broader sense (as opposed to the specific 737-MAX case) is there is still considerable debate on who should have ultimate control (human or software) of the aircraft when making critical decisions. I remember reading an article a while back that made it seem like Boeing and Airbus were on different sides of this spectrum in basic design philosophy, but I don't know how true that is.

It's like any good technical religious debate: both sides have good points and the ideal that they're both working toward is somewhere in the middle.

Airbus has backed off trying automate everything (starting with AF 296 in 1988) and, as this MCAS debacle has demonstrated, Boeing is happy to add serious automation. The days of the "pilot's plane" are long over. Now it's ever thinnner shades of gray.

I believe this problem will become important for more and more industries like medicine, mobility with further advances of AI there.

Best quote in the excellent article:

“You can’t get a degree in how to fly to the moon,” says Dana Densmore, who joined the lab in 1965 and became a control supervi-sor for the lunar-lander software. “You had to get people who know how to think, who are creative and alert. It was all in-vented on the spot.”

She certainly sounds interesting.

https://en.wikipedia.org/wiki/Cell_16

Kind of off topic, but for those interested, this reminded me of the public repo on github which has the original guidance computer source code for the command and lunar modules[0].

[0] https://github.com/chrislgarry/Apollo-11

Went looking for some smart-ass pull request (tongue in cheek) and was gladly surprised there were some! Alas, just proof reading changes and such.

This is the best description of AGC I have seen so far, from hardware design to mission software, layer by layer: "The Ultimate Apollo Guidance Computer Talk" https://www.youtube.com/watch?v=xx7Lfh5SKUQ

If anyone wants to learn more about the Apollo Guidance Computer, this talk[1] from 34C3 will teach you everything you wanted to know and more.

The AGC had some really fascinating features for the time, including a cooperative multitasking system, interrupt driven IO, and even a virtual machine for doing linear algebra!

[1]https://youtu.be/xx7Lfh5SKUQ

I knew a retired couple near where I lived growing up who, I found out later, were software programmers on the Apollo Program. Quite possibly worked in that lab. The husband passed away soon afterwards, and I never got around to asking the wife about her experiences (they both worked for the program!). But it was really cool reading about how the core software was written.

I've been listening to "13 minutes to the Moon" from the BBC recently. https://www.bbc.com/news/science-environment-48232627

One part jumped out at me as brilliant especially for the time: when the computer was overloaded with tasks during the decent, it switched off all non-essential extras like screen displays in order to free up processing power. After that it shed even more to focus on only plotting the landing!

A few weeks back they said it was due to the delta h discrepancy, and Buzz did nothing wrong. This week they were saying that it was down to the docking rendezvous system being turned on, which had not been done before.

I don't know if they came up with new information and forget to redo that episode.

The delta-h discrepancy was a mismatch between the actual height above the moon, and what the navigation system expected, I believe Mission Control took over that.

I feel like the programme murkied the waters.

Fastcompany published a similar article about the 4th crew member of Apollo 11 a few weeks ago. I shared it on HN but it did not get any traction.

https://news.ycombinator.com/item?id=20173672

Do yourself a favor: head to a newsstand or Starbucks and buy a copy of today’s WSJ. This special insert on the Apollo mission is well worth it.

It's nice to hear the Don Eyles story; I haven't heard it before, having only seen stories about Margaret Hamilton and her role.

The article mentions the overload on the computer was caused by "a mismatched power supply to a radar" triggering too many calculations. Any idea how or why the power supply was mismatched ? Luckily software saves the day but what was the root cause of the hardware fault ? Was that a case of a mismanaged change in configuration ?

If I remember correctly, two AC power sources were required to be in phase for measurements of the antenna direction to work correctly, but weren't actually specced as such, and thus not built that way, and randomly ended up so out of sync that it caused problems on that flight.

EDIT: this article has more detail https://arstechnica.com/science/2019/07/no-a-checklist-error...

"The Charming Genius of the Apollo Guidance Computer" is a wonderful talk: https://www.youtube.com/watch?v=xY45YE7ggng

The AGC implemented a virtual machine to run higher-level computational routines. The whole architecture is really interesting.

Eyles has a memoir out: info available here. (The title is "Sunburst and Luminary", which were the names for LM software builds for earth-orbit testing and lunar landings, respectively.)

http://www.sunburstandluminary.com/SLhome.html

> When it came to hiring women for engineering or management, NASA “had a few women, and they kept them hidden,” says Ms. Densmore. “At the lab it was very different,” and there were opportunities for women.

Quite amazing what an important role women like Margaret Hamilton played in the Apollo project, given how hard it is nowadays to get more women into engineering positions. You may remember the famous picture of Hamilton next to the stack of code she and her team wrote for Apollo: see https://en.m.wikipedia.org/wiki/Margaret_Hamilton_(software_...

Love the code comments referenced in the article

This contains more technical information behind the 1202 alarms [0]. It is an awesome read.

[0] https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html

Relevant text:

So what was happening during Apollo 11, as I recall, was that repeated jobs to process rendezvous radar data (that of course were not really there) were scheduled because a misconfiguration of the radar switches. Thus, the core sets got filled up and a 1202 alarm was generated. The 1201 that came later in the landing was because the scheduling request that caused the actual overflow was one that had requested a VAC area.

Quite fascinating that these alarms automatically rebooted computer and picked up essential jobs exactly where they had left off.

> This contains more technical information behind the 1202 alarms ..

“You also have to remember that, long before Bill Gates, we had developed a real-time multi-tasking operating system.”

“When a job was to be scheduled, a call would be made to the appropriate executive routine - sort of like a DOS call today”

edit: I thought this was hysterical, on so many levels, just imagine the scenario if Microsoft had got the contract for the AGC.

> MIT received the first major Apollo contract, the only one awarded to a university, and the only one given without competitive bidding.

Do these kinds of mission-critical components generally go to the lowest bidder?

I think bidding government contracts is a lot more complicated than just picking the cheapest option. There's a lot of other dimensions of validation that bids move through other than cost.

For something like this, I'm sure there is stringent engineering validation. I know some initial NN research was part of a larger government project of funding research around stabilizing missiles in flight for ICBMs that was not cheap.

Competitive bid doesn't have to mean lowest bidder. I've put tons of (non-government) projects out for bid because you're usually expected to do so to keep everyone relatively honest. I imagine the MIT Instrumentation Lab were the only people with the inertial guidance expertise at that time.

"Competitive bid doesn't have to mean lowest bidder"

No you have to wine and dine the right people, and line a job up for them when they retire.

Which gets someone thrown in jail or a lot of fines these days. Not that it doesn't happen, but it's a lot harder to hide now than 20+ years ago.

I was perhaps being slightly misanthropic. But the game keepers always seem to turn poacher. High ranking military bods end up with defence contractors. Financial regulators end up with working at banks (and visa versa), and politicians seem to end up getting various well paid consulting jobs. I suppose you could argue there are good reasons for it, I cant believe on some level these public servants aren't thinking of their career after retirement.

It depends on the contract. Some contracts do go to the lowest bidder (the technical term that the contracting folks use is something like "lowest bid meeting technical requirements"). For those sorts of bids, it's extra important that the requirements that the government puts out are well written and thorough.

Some procurements use the "best value" criteria. I think these contracts are more likely to be protested and have lawsuits filed by the losers. It's easy to say "I deserve this contract because $2 billion < $3 billion". Best value can be a bit more subjective.

https://www.youtube.com/watch?v=cZlLmESPWyw

MIT was probably most knowledgeable about programming computers back then that anywhere else in the world.

Neat story but is the software really hidden? I feel like I have heard this story many times. It was one of the many feats of the Apollo program, I don't think I would consider it hidden.

As a casual space program buff, I thought the same. But I think most people do not realize how sophisticated this early computer and software were, and how important to the mission. The article emphasizes the well-calculated trust that the engineers and astronauts put into this first computer-controlled spaceflight operation.

Yeah, racist software

Reply on news.ycombinator.com