Margaret Hamilton, lead software engineer, Project Apollo | Hacker News Reader

Margaret Hamilton, lead software engineer, Project Apollo

threefingeredfox.net

485 pointsadrianhoward93 comments

Yes, she was one of the early formal-methods people. I met her and Saydeen Zeldin back when I was doing proof of correctness work. They had a company called Higher Order Software, which promoted an extreme form of waterfall design using formal methods. This is appropriate for avionics but didn't catch on. Proof of correctness was just too slow back then.

Incidentally, when you post articles like this, please don't use a title that makes it sound like an obituary.

It's still too slow today for all but the most security critical applications.

Fortunately there is a lot of work in progress to improve the situation.

It's not too slow, it just takes discipline that most developers/companies don't have, primarily because the tools aren't widely available.

What do you mean by "not widely available"? Just as examples, Coq and Idris are both free:

https://coq.inria.fr

http://www.idris-lang.org/

Did you mean they're not in wide use?

Just too languages being available != "tools".

Where are the toolchains, IDEs, libs, etc for those?

I'm trying to install and use Idris as we speak. It seems to require learning two other languages (Haskell, PowerShell) first.

PowerShell? That's a new one....

I haven't used Idris, but as a proof assistant, Coq is one of the best computer games I know[1]. As a programming language, it's a very good proof assistant.[2]

[1] Really! It's fun. I've found the resulting proofs impossible to read, though. The closest description I can make is that they're programs for a stack machine; reading those is not a skill I've developed.

[2] All of the dependently typed languages, like Coq, that I've seen have not been very good programming languages, in my opinion. Except for ATS, which I don't think is a good programming language for different reasons. But I really like it, anyway.

> All of the dependently typed languages, like Coq, that I've seen have not been very good programming languages, in my opinion.

Yeah, that's why I'm looking at Idris - it seems like the only one that's oriented towards practical programming. I did eventually get it installed, but it's tough going doing anything with it when you don't know Haskell tbh.

> Idris - it seems like the only one that's oriented towards practical programming

Whether you like ATS or not, it's very oriented towards practical programming. Also, ATS mostly looks bad IMO because of the style the author uses.

I still haven't fully formed an opinion on ATS though. Idris does seem really cool and stuff like enforcing time/space usage through the type system excites me.

I'm not sure whether my issue with ATS is the style of the author or its type system, although I suspect the former.

ATS is based on ML (the author started with Dependent ML), but the code has a very imperative feel, leading to statements like "let () = expr() in ..." and "let x = ... in ()".

The stuff at http://www.ats-lang.org/Examples.html is just hideous, by the way.

I agree with you, though I think it is because of the authors style. Check out some of Chris Done's ats code examples in a more appealing style:

https://gist.github.com/chrisdone/c23251e8b975dc805876

Here's just a small translation of the ats examples[0] page.

----------------- Copying File ----------------

Authors version:

    fun
    fcopy (
      inp: FILEref
    , out: FILEref
    ) : void = let
      val c = fileref_getc (inp)
    in
      if c >= 0 then let
        val () = fileref_putc (out, c) in fcopy (inp, out)
      end // end of [if]
    end (* end of [fcopy] *)

----------------- Naieve Fibonacci ----------------

My version:

    fun fcopy ( 
      inp: FILEref,
      out: FILEref
    ) : void = let
      val c = fileref_getc (inp)
    in
      if c >= 0 then let
        val () = fileref_putc (out, c) in fcopy (inp, out)
      end 
    end

Authors version:

    fun
    fib (
      n: int
    ) : int =
      if n >= 2 then fib (n-2) + fib (n-1) else n

My Version (not sure if valid!):

    fun fib ( n: int ) : 
      int = if n >= 2 then fib (n-2) + fib (n-1) else n

----------------- Fast Fibonacci ----------------

Authors Version:

    fun
    fibc (
      n: int
    ) : int = let
      fun loop (n: int, f0: int, f1: int) =
        if n > 0 then loop (n-1, f1, f0+f1) else f0
      // end of [loop]
    in
      loop (n, 0, 1)
    end // end of [fibc]

My Version:

    fun fibc ( n: int ) : int = 
      let
        fun loop (n: int, f0: int, f1: int) =
          if n > 0 then loop (n-1, f1, f0+f1) else f0
      in
      loop (n, 0, 1)
    end

0: http://www.ats-lang.org/Examples.html

Having learned and put into practice formal proofs of correctness at CMU, I can say that it has fundamentally improved the discipline with which I write production code. The undercurrent of "Is this provably correct" is still there 30 years later, and it produces better code than the more obvious "Will this work here". The cost in production time is small compared to the later revision, refactor and maintenance issues.

I wonder if we are talking about the same things. Do you use Hoare logic or interactive proof assistants when putting "nto practice formal proofs of correctness", and find that that lowers your software engineering costs? I would be surprised if that was the case.

I worked with Margaret and Saydeen at Higher Order Software for my first job out of college, but I didn't get to work on anything as glamorous as Apollo. We did mostly government contracts; I was there for two years before I decided it was time for grad school.

OT: I just said to myself, that cannot be "the Peter Norvig" here on Hackernews and I checked the profile. And Lo and behold ...

Is HOS what it seems? I read about it years ago in "System Design from Provably Correct Constructs". It seems like it should revolutionize software development (and incidentally put most programmers out of business) but that it somehow just never gets "traction" beyond certain markets/customers?

The first time -- and still one of the few times -- I ever walked out of a meeting because the BS level seemed too high was when I visited Higher Order Software in the 1980s.

Can you please share details of what caused you to walk out? There is a lot of curiosity around USL and the toolchain she and her team have built, but my Google-fu has failed to come up with descriptions of practical experience with them. I'm struggling to decipher how different it is from UML, or what it is solving that functional approaches are not already working to address.

It was 30ish years ago, so I don't recall details. But it seemed overhyped. The usual problem with tools that do a great job of generate code is that they do a great job of generating CERTAIN KINDS of code, but are unhelpful or even counterproductive with other parts of your development task. Certainly this was true in the 1980s, and back then the problem was exacerbated in that there weren't graceful ways to combine what was generated with what you had to write or modify by hand.

Okay but was it BS, or did it just sound like BS and actually it's important and useful (if a bit rough and early)? That's my main curiosity.

I find it a bit... sad? disturbing? that the only reason women were so involved in computing and programming in the early years was because programming and software were seen as less important jobs than the "software planner" jobs, where the "coder" was just a flunkie, something of a software custodian. They were considered clerical jobs, like data entry or a secretary.

This old article (2009) has a lot more details on this old sociological phenemonon:

http://homes.soic.indiana.edu/nensmeng/files/ensmenger-gende...

The author has a whole blog about this topic:

http://thecomputerboys.com/?p=239

Definitely sad, but I'd say the disturbing part is how fast they got disappeared once software development became "a real job".

I started programming before I was a teenager, in part because my mother was a programmer. At that time, 30+ years ago, women were not programmers because it was "less important". You didn't see men taking programming jobs as a "Consolidation" for some better career.

My first two programming teachers- in college and in high school- were both women who were also professional programmers at previous jobs.

It was a prestigeous job, as the computer was seen to clearly be the herald of a new age. The personal computer revolution was really significant. It was a job for smart, mathematically inclined people.

I think programming is seen as a lower status job now than it was then. Now programmers are subservient to ignorant business people (When the reality is, its much easier for use to learn business than them to program.)

I think in the past women were more likely to be programmers because the profession didn't have a gender stigma. Because men were more likely to have picked out a profession, and women who entered college without a specific degree plan were more likely to pick this one up.

And frankly, I've never seen a problem with women programmers. I've worked with women programmers at every job where there were more than 10 programmers, and they haven't been seen as deserving, or treated with less respect... this goes back to the 1980s.

I would like you to back up your assertion that the "only reason" they were involved was low status. That paper you linked to is fallacious beyond belief. You may disagree with my anecdotes below, but he's merely giving us his spin on an article. (So he's not only anecdotal, but derivative.)

> 30+ years ago

That's barely in the 1980's or late 1970's, well after the events being described in the articles I mentioned.

Can you contest that Von Neumann considered programming a clerical job?

> That paper you linked to is fallacious beyond belief.

Well, I suppose I would have to read his citations too, to check if they were all lies as well.

If you're going to make a list of people that John von Neumann looked down on, it's going to be rather longer than just women (try anyone who wasn't as smart as he was, which is just about everyone).

Brilliant scientist. World-class ego.

i wish i worked in the places you worked. the ignorance in the places i have worked is utterly gobsmacking.

>because programming and software were seen as less important jobs than the "software planner" jobs,

It's not like STEM software today like the codebase of ATLAS for example, is give much praise in the public eye. It's simply overshadowed by the larger more ambitious project that it was part of. This was the case back then as well.

Definitely a role model.

Her site has a pretty neat article called "Inside development before the fact":

> Today's traditional system engineering and software development environments support their users in "fixing wrong things up" rather than in "doing them right in the first place". Things happen too late, if at all. Systems are of diminished quality and an unthinkable amount of dollars is wasted. This becomes apparent when analyzing the major problems of system engineering and software development.

Also a cool paper she wrote:

> The key to software reliability is to design, develop, and manage software with a formalized methodology which can be used by computer scientists and applications engineers to describe and communicate interfaces between systems. These interfaces include: software to software; software to other systems; software to management; as well as discipline to discipline within the complete software development process. The formal methodology of Higher Order Software (HOS), specifically aimed toward large-scale multiprogrammed/multiprocessor systems, is dedicated to systems reliability. With six axioms as the basis, a given system and all of its interfaces is defined as if it were one complete and consistent computable system. Some of the derived theorems provide for: reconfiguration of real-time multiprogrammed processes, communication between functions, and prevention of data and timing conflicts.

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=170233...

And some NASA work related to HIOS: http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/1975001...

Besides the great photo and backstory, I liked this snippet:

> She was all of 31 when the Apollo 11 lunar module landed on the moon, running her code. (Apollo 11 was able to land at all only because she designed the software robustly enough to handle buffer overflows and cycle-stealing.)

You could read that paragraph in several ways. Back then, accomplishing something like that at 31 seems precocious. Today, in the hype about 18-year-olds becoming millionare-startup founders, she sounds like a late bloomer of a programmer.

Here's another great photo (http://i.imgur.com/2jJZwBZ.jpg) which was posted on the TIL thread a few days ago (http://www.reddit.com/r/todayilearned/comments/2oskna/til_th...).

See, I see this and the other photo of her as conveying the message, "You don't have to look like the stereotypical 'NASA nerd,' the white guy with the white button-down shirt and the pocket protector full of pens, in order to make a contribution. Some of the biggest contributions were made by people who didn't fall into that stereotype."

At least she didn't make a fucking app.

She made an app that landed spaceships?

Feats of engineering, while technically within the definition of "application" like virtually every other artifact of deliberate human action, rarely actually get described as such.

More detail on the cycle-stealing bit, which caused an alarm during the Apollo 11 lunar descent, which the AGC's priority scheduling allowed it to recover from: http://en.wikipedia.org/wiki/Apollo_Guidance_Computer#PGNCS_...

Wasn't that the same alarm that was randomly thrown at the astronaughts in training and they made the wrong choice then and learned to make the right choice on the moon?

I will look up the story in Kranz's autobiography when I get home tonight.

I do believe it was. The Simsups threw a 1201 alarm at them, and the controllers got confused, and Kranz called an abort. Afterwards, they told him, "It wasn't an abort, you should have continued the landing...I thought you guys had it, but then you went off the rails and called for an abort...You violated the fundamental rule of Mission Control: You must have two cues before aborting. You called for an abort with only one! " As a result, one of the controllers went more into depth on program alarm codes, and they put a rule in dealing with which alarm codes would be grounds for an abort. Neither 1201 nor 1202 were on that list. They saw both those alarm codes during Eagle's descent, and were able to say, "We're Go on this alarm...we're still Go, same type, it's a Go." And the rest was history.

Some years ago, there was a video about the moon landing in Washington DC airport (it was at a Smithsonian exhibit or store or some such). It included video of the control room when the alarm went off during descent. There was silence and blank faces - nobody knew what to do.

But by the time the alarm went off the second time on descent, it was completely different. Someone - whoever was in overall charge - called a role (something like "descent control officer") and that person responded "go", all within one second.

But, yes, it wasn't a problem because the priority scheduling recovered from this problem, and that recovery was a really big deal.

Steve Bales with feedback from Jack Garman (Gar-Flash).

The amazing thing is that Garman was the only one to have a quick reference of the various error (numeric) codes.

> Some years ago, there was a video about the moon landing in Washington DC airport (it was at a Smithsonian exhibit or store or some such). It included video of the control room when the alarm went off during descent. There was silence and blank faces - nobody knew what to do.

You have to be careful interpreting something like that. Without the backstory, you run the risk of projecting your own emotions onto the events.

Mission Control was actually not very worried about the program alarms. According to Gene Kranz in his memoirs, the white team had been fed a program alarm during a pre-flight simulation. In fact, it was the last simulation they ran through before launch, so it was fresh in their minds.

During the simulation, Mission Control incorrectly called an abort. During the debrief, the instructors told them that the program alarm was non-fatal, and they should have continued to a lunar landing.

During the actual lunar landing, Mission Control was astonished to run into the same scenario. It was like running through the simulation a second time.

> But by the time the alarm went off the second time on descent, it was completely different. Someone - whoever was in overall charge - called a role (something like "descent control officer") and that person responded "go", all within one second.

Neil Armstrong did have to ask twice, but he did get a response from Mission Control on the first alarm. Mission Control did not ignore the first alarm.

The delay was caused by having to consult a list of fatal and non-fatal program alarms. Once they had that list, it was easy to call a Go on subsequent program alarms. However, they couldn't just give a Go to the first alarm without checking the list.

Armstrong and Aldrin hadn't participated in that earlier simulation, so they were probably a lot more worried than Mission Control was!

Here is the transcript with references to Kranz's book: http://www.hq.nasa.gov/alsj/a11/a11.landing.html

Considering she didn't learn how to program a computer until her 20s, and there were no shoulders of giants (sorry, frameworks and operating systems) to stand upon, I'd say it's a more impressive feat.

And no Internet, no Stackoverflow, no "common knowledge".

Incredible.

It looks like she is currently CEO of a company that sells a formal-methods language (and accompanying IDE) called 001. It looks interesting - has anyone here worked with it before?

http://www.htius.com/

More details about her work on Apollo Guidance Computer:

Apollo Guidance Computer History Project, Margaret Hamilton's introduction:

http://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/a...

"Many of the things I was intrigued by had to do with how to make the mission software safe and reliable. And one of the things I remember trying very hard to do was to get permission to be able to put more error detection and recovery into the software. So that if the astronaut made a mistake, the software would come back and say "You can't do that." But we were forbidden to put that software in because it was more software to debug, to work with. So one of the things that we were really worried about is what if the astronaut made a mistake -- We were also told that the astronauts would never make any mistakes, because they were trained never to make mistakes. (Laughter)

So we were very worried that what if the astronaut, during mid-course, would select pre-launch, for example? Never would happen, they said. Never would happen. (Laughter) It happened."

She also mentions her companies:

"So since that time, the theory has evolved and now I actually lost the first company to venture capital people, and started a second company called Hamilton Technologies."

The link to the second:

http://www.htius.com/

The wikipedia article on her notes she's credited (by NASA) with coining the term "software engineering".

http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%...

Interesting, and something I didn't know. The cited source unfortunately doesn't elaborate on the context. The chronology does seem about right. The first in-print use of the term I can find is from 1969, when it was used as the title for the proceedings of an October 1968 NATO scientific conference: http://dl.acm.org/citation.cfm?id=1102020. Given the NASA/NATO personnel overlap at the time, it's certainly plausible that they got it via NASA.

Good to know, thanks!

She was briefly mentioned in Steven Levy's "Hackers", as an MIT computer user who's program crashed after the "Midnight Wiring Society" changed how the computer worked.

Also, she was interviewed in "Moon Machines" episode three, about the nav computer.

http://en.wikipedia.org/wiki/Moon_Machines

Awesome. She's a role model for girls, which our field badly needs. The client list on her company's website is pretty long and substantial.

She's a role model for everybody!

Definitely.

Sure, but we men already have plenty of computing role models. This is a welcome change.

Both female and male computer scientists need good female role models, and for the same reasons. In my own life I've been very fortunate to have been influenced by my mother (who taught me to program at an early age), by a grade school computing teacher (who also encouraged me to learn programming), and by my wife (a professional software engineer, among other things). I like to think that my wife's professional career has given my kids some similar perspective - even if only to change their unconscious assumptions about what women can do (which is everything, since my wife has been a writer, a software engineer, a business analyst, and a small business owner - and that's just what she's done professionally). Women needs to believe that they're capable of everything, and men need to believe that women are capable of everything, too. The same goes for men about men, and women about men. Feminism isn't about replacing patriarchy with matriarchy. It's about egalitarianism for all genders. That's all I meant.

I just copy and pasted the article and attached the photo to an email to my 1 year old daughter. I have an email account for her for just this kind of thing. I can only hope she's as inspired by it when she grows up and reads the emails as I hope she will be :)

That's a super idea! I'm going to do it.

You just brought my fiancee and I to tears. What a brilliant idea.

What a truly inspiring human being. I can only dream of aspiring to her levels of contribution.

Check out some of the Apollo 11 code for the Lunar Module's (LM) Apollo Guidance Computer (AGC). It's just awesome browsing through it, reading the comments, and thinking about the zeitgeist of being on a team that was working on something of that world-altering magnitude.

Code Library:

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

KALMAN_FILTER.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

LAMBERT_AIMPOINT_GUIDANCE.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

LANDING_ANALOG_DISPLAYS.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

LUNAR_AND_SOLAR_EPHEMERIDES_SUBROUTINES.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

LUNAR_LANDING_GUIDANCE_EQUATIONS.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

- - -

Incredible quote:

"There was no second chance. We all knew that. We took our work very seriously, but we were young, many of us in our 20s. Coming up with new ideas was an adventure. Dedication and commitment were a given. Mutual respect was across the board. Because software was a mystery, a black box, upper management gave us total freedom and trust. We had to find a way and we did. Looking back, we were the luckiest people in the world; there was no choice but to be pioneers; no time to be beginners." - Margaret Hamilton

- - -

Edit: OK, this is interesting. Note the filename. The filename and comments suggest that it was for driving keyboard and information display...

PINBALL_GAME_BUTTONS_AND_LIGHTS.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Lum...

A few of the code modules that mention Hamilton:

Artemis072/ASSEMBLY_AND_OPERATION_INFORMATION.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Art...

Artemis072/SERVICER207.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Art...

Comanche055/CONTRACT_AND_APPROVALS.agc

https://code.google.com/p/virtualagc/source/browse/trunk/Com...

---

An overview of the code:

http://www.ibiblio.org/apollo/Colossus.html

I'm having a hard time believing that's a working kalman filter implementation in assembly in 70 odd lines!

Check this out... here are the two source code pages (digitized from original printouts) for the Kalman filter, dated July 14, 1969!

KALMAN FILTER USER'S PAGE NO. 1

http://www.ibiblio.org/apollo/ScansForConversion/Luminary099...

KALMAN FILTER USER'S PAGE NO. 2

http://www.ibiblio.org/apollo/ScansForConversion/Luminary099...

According to wikipedia "The onboard guidance software used a Kalman filter to merge new data with past position measurements to produce an optimal position estimate for the spacecraft." https://en.wikipedia.org/wiki/Apollo_PGNCS

In turn the main source of navigation information came from mission control using least squares. A source on that would be nice too!

after having worked on a kalman implementation in processing, I had the same gut reaction.

The scalar case is not so bad; typically the filter is taught using the vector-input/vector-state case, which is more complex.

I can't make heads or tails of the assembler listing either. The specific thing I would look for in the assembler code is the division operation which is required to find the Kalman gain. I don't see it.

But, the anecdote about the Kalman filters use in this setting is well-known. (E.g., search for "Apollo" in http://ieeecss.org/CSM/library/2010/june10/11-HistoricalPers...)

The amazing thing from my point of view is not the implementation. It is rather that the theory, which was quite novel at the time, was only understood and published in 1960/61. To have it used in a flight system, and indeed a manned flight system, only 8 years later is really incredible.

A lot of people will find it hard believing how smaller code can be in assembly than in our supposedly higher level languages, e.g, look at this example of quicksort, http://en.wikibooks.org/wiki/Algorithm_Implementation/Sortin... go down to the C example below and note that it's longer.

indeed, its the kind of thing where you hope to see such epic code comments like this:

// and the following section is where we attempt to land on the Moon

When this story was posted yesterday [1], the comment about UDL sparked my interest. That led me to Hamilton'sDevelopment Before the Fact methodology [2]. I find DBTF an appealing approach to reliable systems, particularly buildings. Because AEC design software is largely focused on automating manual steps rather than the generation of systems.

[1] https://news.ycombinator.com/item?id=8731711

[2] http://www.htius.com/Articles/Inside_DBTF.htm

> I suppose today’s kids are ho-hum about these recoveries of memory, but I think they’re pretty neat.

No, we think they're pretty neat too :-)

In early computer days a fair fraction of the programmers were women. They migrated from earlier computer jobs such as keypunch operators, switchboard wirers (some early computers had programable switchboards) and mechanical calculator clerks. Until mid 1940s "computer" meant the third category- a person who operated mechanical calculators.

I hope this serves as tangible evidence that women do have their place in tech. I feel it's a shame that we believe (rightfully or not) that we need extra coding schools for females.

My take on this: If more women would be into (information) technology, we would have cheap holidays on Mars!

Margaret's role is a role model.

what a great read. she looks so happy! she is very pretty as well!

Her with a hardcopy of her code.

http://i.imgur.com/bxyWmuA.jpg

shes brilliant and gorgeous. I don't see anything wrong with that

Reply on news.ycombinator.com