Carmack: Armadillo Aerospace in “hibernation mode” | Hacker News Reader

Carmack: Armadillo Aerospace in “hibernation mode”

newspacejournal.com

126 pointspvarangot54 comments

The part I find fascinating about Carmack's comments are the 'sliding into NASA mode'.

NASA, and contractors working for them, has this culture of how these things are done. And if you hire people with experience, then they probably got that experience in a NASA shop, and when you get enough of them your culture tends to revert to what they are comfortable with, and that leads it into the same "zone" as NASA.

I see that in startups and small companies, where the employees bring their 'imprinted' culture with them, whether its a power point heavy culture from Oracle or a 'lets try this' sort of culture from Google, or the "secret sauce" culture from Apple.

I was convinced this was going to be what happened to SpaceX but they seem to be resisting assimilation so far. I hope it can last as I think they need to break some of those cultural binds to get to where they want to be.

I hope John gets a chance to write up the good and bad aspects of how things were done to help future space company employees avoid the bad ones.

I'm also curious to see how SpaceX makes the transition to producing man-rated production vehicles. When someone at NASA says "prove this is safe", that forces a lot of the structure and painstaking process back in.

SpaceX's rockets are already "man rated" by default, or is designed to be man-rated.

True, I guess the Falcon-1 was their non-man-rated testbed.

Elon wants his rocket to be man-rated so he can ride it to Mars. The fact that NASA needs to get the ISS is just gravy.

I've seen that phenomenon a couple times with people from Microsoft; I wonder if it's true of Googlers in some other way, too.

I'm curious, what behaviors (for good or bad) do you see that people of Microsoft bring to other companies?

In my experience

1. Put all the code into the database

2. Everything is or should be xml

What the hell part of Microsoft were they from? And were you able to get the answer to that question before they were shot?

Given the amount guys like this[1] that work at Microsoft love XML, it sounds pretty reasonable. Actually liking the XML docblocks though is taking it a bit too far. There's a lot of things I dislike about Java, but one thing they got right² (and almost everyone else used after) is their documentation blocks. When using Visual Studio I have to make the syntax color for XML elements of the blocks I generate super contrasting (usually really dark) with the documentation inside just to read it without making my head hurt.

> During the years I’ve seen a lot of different ways of commenting code – some good and some bad. My personal favourite way of commenting code is by using the XML comment feature of C# and VB.NET, but the important part of commenting code is not how you do it, but what you write in them.

[1] http://madskristensen.net/post/The-what-why-and-how-of-XML-c...

² Java may not be the first, but it's the first language I saw to use that style

It's funny to read about the Microsoft developer culture now vs. when I left fifteen years ago.

MSFT was the PayPal, Google, etc. of our generation in Seattle. It spewed out startups of FYIV people in a short period of time.

At the time, it was very disruptive to hear that "ex-Microsoft engineers" were entering a market. There were a few things that seemed pretty common in the groups with which I had contact. (SQL, LAN, VC++, etc.)

1) QA as a first class function. SDETs were awesome, and they tried to staff projects with about two for each SDE. Testing tools were automated and accounted for huge amounts of code. Compatibility labs were huge. Breaking the build would be just short of a flogging offense.

2) A focus on maximizing productivity. Spend the money on the desk that can be raised or lowered. Spend the money on the newest, latest computers, and give them to anyone who asks, as many as they think they need. Locate beverages very close to everyone and subsidize outside businesses to provide food services. Give everyone a window office. It made it a lot easier to stay there from 8am to whenever. This also led to the diaspora of startups.

3) Scary-ass yelling matches in the hallways. People who had always felt they were smarter than the jocks and popular people in their youth were suddenly capable of establishing dominance by insulting other people, their intellects, and especially their work.

4) Readable code over clever code. This led to a lot of typedefs like LPCSTWCSRARSARTARSTA[1] and Hungarian notation[2]. So if you see MyMethod(lpszName); somewhere, you'll know where that came from.

People would leave after vesting (sometimes before and sometimes long after) with the idea of creating a product of their own with the good things that Microsoft taught them and without the bad things that they hated.

Very much like when we start thinking about having kids with the intention of not making the mistakes our parents did.

[1] Yes, Colemak.

[2] https://en.wikipedia.org/wiki/Hungarian_notation

Some of the coding style from Google carries over.

// TODO(pjscott): provide a concrete example of this.

Theres a lot of great articles on Scaled Composites (both before and after they won the X prize) that describe the culture and what it was like to work there.

http://www.wired.com/wired/archive/11.07/space.html

They seem very different from the "NASA" mentality, and much younger and more startup-focused. Very (extremely!) small team, given what they accomplished.

They also lost some people to some kind of test accident. I'm not saying this to point fingers, but to remind people that in aerospace, you're not dealing with. "Oops, I ran the wrong build configuration. I guess I'll just restart."

http://www.knightsarrow.com/rockets/scaled-composites-accide...

To be fair Carmack also comes with his own cultural baggage from Id Software. Wanting to keep the team small, not setting hard deadlines, using a new generation of the engine for each game etc.

It worked, didn't it?

Not really; The development time seems to be getting longer. Quake/Doom 1 took a ~year. Doom 3 was 3 years, and now Rage the latest title took 7 (!) years.

Right, but look at the team size as that happened. I'd posit that moving away from the team structure he advocated is part of the cause of those delays.

I am not entirely sure how good Carmack was for the company—the tech engine yes, as the director of a corporation, not so sure.

Agreed. It's obvious now that losing Romero was a tremendous blow to the company, creatively.

(Was not obvious back when it happened though, at least not to me.)

And look at the scope.

What do you mean? All their games were aimed for the same ~12 hour play duration. Rage was no exception.

The amount of content you have to produce for 12 hours of Rage is much greater than for 12 hours of Doom

There's also the dynamic of feeling the need to do more, having learned from previous mistakes.

My employer's first hit game was shipped in 5 weeks by one full-time programmer and one full-time artist. It took three years to become what it is now.

Now, when we start a new game, we worry about over-the-air downloads and updates. We think about monetization. We work with testing partners. We need privacy policies. We need Facebook integration. We need localization (English, French, Chinese-Simplified, Chinese-Traditional, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish). And on and on ...

None of this is explained by a growing team, nor by outsiders brining in their biases. Instead, we feel we need to do more to avoid the mistakes we made in the past.

Is this a maladaptive, or is it a realization of the scope of the problems we're trying to solve?

No one dies if we ship a brutal crash and data corruption bug. I love rapid prototyping, but I think new-space is realizing that rapid development means $100M, not $10M invested.

An interesting aspect is that NASA gets criticized from both sides. Here Carmack is basically criticizing NASA for not taking enough risks, being too cautious in the meticulous engineering instead of whipping out a saw and trying something. But whenever something goes wrong and there's an inquiry, the inquiry always blames NASA for being too seat-of-the-pants, not doing enough proper engineering and analysis of safety margins, not following procedures to the letter, etc., etc.

In American culture, the death of one person is typically considered unacceptable, and "we will do everything to ensure it never happens again." To date, NASA has flown many more manned missions than its private competitors. Most unmanned missions can have their cargo and vehicle replaced (albeit at great cost), but from the first-world view of "every life is precious," the same cannot be said of manned missions. So there is a fundamental, qualitative difference between the two types of missions and organizations. You can well imagine what will happen each and every time a human life is lost on a private space mission.

Will we ever consider it the cost of doing business? Probably not, except for military missions--and in the context of going to Mars, we might see more of those.

"Here Carmack is basically criticizing NASA for not taking enough risks [...]"

I understand your point, but I think it's important to take this in the context of what type of risk is being discussed (i.e., payload-only).

"We got to the point, where, we were scraping by, I mean, we had an operating profit, but it was doing contract work for other people, and I reached the conclusion that we just weren't going to get where we needed to go with that.

There's this tempting thought, that, if you work for contracts people, and you pick the right contracts, you'll be developing things that you wanted to develop anyways, that it will help you towards your goal. [...] It wasn't working for us. We were keeping the lights on, we were building rockets, but they weren't the tasks, the things that we needed to do."

That was about 2 years ago. They got side-tracked. They moved away from their core competency of what they needed to do, and started taking on outside work for what they thought would supplement their efforts, but it ended up moving them away from where they needed to be, and the added process requirements, as John said, "[...] it definitely slowed us down."

Clearly, when you go into human spaceflight mode, you are going to want to have a lot of well-defined, rigorous process in place. However, they were dealing in payload-only scenarios at this point, with entities that knew the "at risk" status of the launches. In John's words:

"They were sort of like, 'at your own risk'... you can put 'em up, you might get your payload back, but there's a good chance we're gonna crash. [...] They knew it was at-risk, and we didn't take their money when we didn't make a successful flight."

Given what they were doing (i.e., payload-only), this is the context in which he was saying that, "[...] NASA loves seeing all this stuff, it gives everybody that's contracting a good set of warm fuzzies to get stuff you hold in your hands on the engineering documents [...]"

I do not know this as fact here, as I am speculating, but it is possible that they were simply in the mode of iterating to their best ability, with the concept of operations being, "Let's rapidly prove what we can do in the payload launch and recovery area, and once we get funding, we can scale up the process and procedure aspects required for human spaceflight."

It would be interesting to hear John's comments on this topic, for sure... but in summary, from the context of his comments, I think his line of reasoning was perhaps, "Why are we taking on all of this contract work and getting slowed down in process for payload-only scenarios, when we could be building, launching, iterating and learning?"

There's a large difference between experimentation and not properly vetting a vehicle containing living things.

I've always admired Carmack's candid insights. Rather than put up a PR facade instead he readily admits that he feels Armadillo stumbled because he took his hands off the wheel, failed to mass produce test rockets, and the staff spent too much energy coordinating with NASA rather than doing R&D. Very helpful pointers for entrepreneurs.

I don't care about these setbacks, I still have hopes for Armidillo and I think Carmack will realize his dream of VTOL rocketry eventually. He's taken the right approach of addressing a hard problem that no one is solving, and he is solving it very methodically. It's only a matter of time before the last obstacles are cleared.

In addition to being candid, it also never ceases to amaze me how normal he comes off, despite being John Carmack! I love the bit about how funding this company has always been a negotiation with his wife.

It felt more bit shifting-the-blame to me than candid (but maybe he does believe it). What happened is John Carmack started an aerospace company thinking that he could parlay his success in writing game engines into shaking up aerospace, but he couldn't. And now he's trying to blame it on the people he hired, instead of admitting he failed. He seems to want to keep alive the fantasy that only if only he had really worked at it, his idea was right all along, and it's only the subordinates tasked with implementing it who failed, not him.

Perhaps it's time to go back to 'volunteer' mode and putting everything under open source. That way dedicated hobbyists might have the opportunity to take what Armadilo to the next level without dealing with NASA's bureaucracy.

Just a thought. I'm sure there's a reason it won't happen though.

That would be nice, but it would only directly be of use to very dedicated hobbyists with a lot (millions/USD) of money to burn on their hobby and lots of time to deal with regulations (and I don't say that in the typically dismissive libertarian way, as I believe FAA and other regulations as they relate to launching rockets of significant size are a good thing to have even if they probably go overboard).

I don't know if they have been dealing with stuff that would be covered by it, but ITAR might be a problem for open sourcing it all.

" (a) Rockets (including but not limited to meteorological and other sounding rockets), bombs, grenades, torpedoes, depth charges, land and naval mines, as well as launchers for such defense articles, and demolition blocks and blasting caps. (See § 121.11.)

http://www.fas.org/spp/starwars/offdocs/itar/p121.htm

They were apparently working on sounding rockets? Not sure.

Would be nice if FAA would set up an experimental rocketry like Experimental Aircraft segment.

I understand realities of our govt paranoia will not allow it happen. It's a shame. Homebrew cars, motorcycle, airplanes etc have a long tradition in the US and we benefited enormously from that. Rockets are a natural next step. Or we won't have Zefram Cochrane ;)

I'm normally against this sort of thing on principle, but I think there is actually a legitimate concern when it comes to rockets. The only real difference between rocket technology and missile technology is intent.

This said, I think we are approaching an era when the publicly available rocket science becomes "good enough" for effective missile use, rendering continued restriction pointless. Currently the North Koreans have some serious trouble getting stuff up there (a single satellite, orbited very poorly), so the restrictions for rockets actually seem effective, but once they've figured it out (and it is only a matter of time), then I don't think there will be much point to continued secrecy. It would kind of be like China trying to keep the silk worm under wraps today. That cat is out of the bag.

I don't know how much rational arguments enter into it.

The U.S. State Department has consistently avoided going to court on ITAR questions, because some of its pronouncements seem clearly unconstitutional. They'll never let it be tested, though.

There was a case where a civilian IMU, or some-such was used in a civilian aircraft and then later used in a military missile. Because of the missile use, it fell under ITAR and the U.S. aerospace firm that was using it in the civilian vehicle was fined. This seems like a retro-active law which is unconstitutional. It was easier for the firm to just pay the fine, rather than litigate.

I'm lazy, and this is anecdotal, but this story seems to be well known in new-space circles.

Well, I would say that rationality and constitutionality are two unrelated issues. It seems quite probable that it is unconstitutional.

The general idea of restricting rocket technology as long as there are "baddies" that still don't have it seems reasonable enough though.

ITAR is a perpetual concern for people working in that field. It leads to a lot of grim jokes, but also, presumably, a lot of self-censorship.

I have a sticky note somewhere in the office that say "Open Source Rocket Science". One day I'll get to put a tick beside it, I'm sure of it.

I think fundamentally, a million dollars a year is just not enough cash for this kind of venture. All hardware is expensive. High performance hardware with an innate chance of failing (rockets, submarines, airplanes) is even more expense. Sales to the government are expensive and SLOW. If you are doing both gov't + hardware, it's going to take some big bucks for relatively low reward. There is a reason few have really "disrupted" one of the major defense contractors. SpaceX is the closest, but even then, that's only 1 of many different products the major defense contractors are building.

Yeah. Some amateur rocketeers used to deride Elon Musk for spending more than $100M, claiming he was "old space", not "new space". But maybe, that's just the scale of the problem.

I suppose progress depends on the unreasonable man, and all that, but if you're too unreasonable, you just hamstring yourself.

I disagree, somewhat. A million dollars a year can be a lot of money. But it's very difficult to constantly maintain spending that money in the most efficient way possible. And all it takes is making one big mistake to lock in the development trajectory to a track that is very inefficient and suddenly slows down with the given budget. At that point it becomes very difficult for most organizations to fix things, because it means abandoning work that the team has already planned for, partially completed, made commitments on, and so forth. The only way to get back to the more efficient model is to make a drastic hard-break and abandon a lot of current work. Though it might be very non-trivial to recover the state of development back to a previous point. Such an effort is also likely to be very demoralizing as well.

The trick is to have someone in a position of power making sure that such "trajectory excursions" either never happen or are easily recoverable from.

I know he says otherwise, but Armadillo never really struck me as a business out to make money. It was more a business to help develop workers and technology. The Carmack Prize, while not a part of AA, I think represents a philosophy in trying to encourage more thinkers into becoming hobbyists and more hobbyists into doing real stuff, which provides an excellent training environment for people to go onto other space companies.

From the keynote:

"But I was disappointed that it didn't turn out better. And I have, you know, a few theories on why it worked out that way.

You know, the point that I try to avoid, is a hubristic point, is that I wasn't as involved with Armadillo in recent years. I had smart guys there that I felt were competent and I respected their directions, but, me not being there left me in the position of not wanting to second-guess the boots on the ground.

When something would come up, and I'd be like... 'I kinda think you should be doing something the other way', but if I'm heads down on software, they're the ones there every day working on things. I didn't feel really justified... I never want to be that manager that's out of touch with what's going on in the engineering, that's just saying, 'Back in my days, we would have done it this way.'

So I let my hands off the wheel there, and we can't make an A/B comparison about how important that may or may not have been, but there were some of the things that I term 'Creeping Professionalism' that came into it, where... in the old days at Armadillo, we were all about, OK, here's an idea, run down to the shop, grab the saws and the welding gear, slap something on the mill and, you know, you've got something next week and it's fabulous, but, as we did more contracting work, and as people got full-time, we would have big engineering drawings, lovely stuff comes off the plotter with all the information about it, we have our revision controls and our document systems, and of course NASA loves seeing all this stuff, it gives everybody that's contracting a good set of warm fuzzies to get stuff you hold in your hands on the engineering documents, to know that we had a preliminary design review... all this makes people feel good when they're paying for engineering, but it definitely slowed us down.

Some of it may also have been the move to full-time employment, where, when we were all a volunteer team, when it was just me paying the bills, and everybody had a full-time job and we came here and we worked on rockets a couple days a week, everybody was really focused on getting the work done. If you know you're gonna be here for 5 hours, or here for 8 hours, there's just not a lot of time, you know, for goofing around.

Actually, we had a little bit of that when more people, we sort of hit a critical level, when we had a couple more volunteers, and I noticed productivity actually dropping, we pared out some of the volunteers, and productivity went back up. So there was certainly some of that effect.

When you get everybody and it's your job, you go in, and you know, you check your mail and browse the web a little bit in the morning rather than going first-thing down into the shop.

I had this thinking that, OK, people are working 20 hours a week part-time, and they've still got a full-time job, we should be able to get 3 times as much stuff done when everybody is full-time, but it didn't work out that way at all, where, when rocketry was people's full-time job, they got other hobbies, and they did other things with their 20 hours a week on it, and then your 40 hours full-time a week is very rarely 40 really good hours for most people when you wind up having all the little things you do at work that are not really work.

So, there's a number of things that may have contributed.

And we're still tantalizingly close.

One of the bad things, the things that we didn't do is we should have made series production, we should have made multiple vehicles at once. And we did this years ago with our modular vehicles, we made parts for 5 of them, and that served us so well. We got to crash every single one of them, but it wasn't that traumatic because we had more pieces there, and we just built another one. We desperately should have done that with these 2 rocket vehicles. That was our critical mistake in the last 2 years, because we should have been able to put more of these together.

'Creeping Performance' was another thing, where, we used to just make everything out of aluminum, it's not like the highest-tech material, but carbon fiber started creeping into our development systems here. We started heat-treating our own aluminum rather than just using thicker aluminum, and all these other things... this is chapter and verse from some of the errors that NASA has done over the years, and it's heart-breaking for me to see my own team following some of these problems."

Nitpick: "serial production", not "serious production".

Thank you. I re-listened... it sounds like he is saying, "series production". I've corrected.

Nitpick #2: Creeping Professionalism

Here's the link to the relevant portion of the Quake Con keynote that is transcribed above (and the submitted article is based on): http://www.twitch.tv/bethesda/b/439369577?t=2h04m56s [10 minutes]

Why not just sell the technology and IP to another company that wants to make use of it?

Because he hopes to be able to return to it one day. I don't think he's in it for the advancement of science or the industry, but rather for the kicks of it. It's his hobby.

If he's got anything unique that could help further the effort to make space travel viable, it'd be nice to see it at least licensed. Armadillo did a lot of work on various launch strategies.

That is very sad, I had lots of hopes for them, and for the rocket racing league too!

Reply on news.ycombinator.com