Comment by gpm | Hacker News Reader

Comment by gpm | Hacker News Reader

Parent Full thread

From a process design/management view I'm not sure you're right. In fact I think you're probably wrong, but there is definitely room for disagreement.

To oversimplify things massively, "how careful do we want to be to avoid testing failures" is a parameter that SpaceX management gets to control. At the extreme careful end everything costs billions of dollars because you're paralyzed analyzing things and double/triple/quadruple/quintuple checking things. At the not at all careful end of things you keep building "complete rockets", pointing them at mars without any testing, and blowing them up. Obviously neither is rational.

They've settled on some parameters that are basically "don't worry that much about test failures, but keep the failures really cheap". They seem to be doing pretty well by doing that. They're experimenting with manufacturing techniques, hiring and training a workforce, building facilities. The prototypes that they are building keep failing, but that looks to be a relatively minor cost considering that the current primary goal is (according to them) to build out the manufacturing processes and make the design easy to manufacture.

Being slightly more careful would undoubtedly reduce the numbers of prototype failures, but would it actually be worth the cost of slowing other things down? Remember that the main cost of this program to SpaceX is engineering salaries, the faster it goes the cheaper it is.

So of course this test didn't go as planned, of course it would be better if it had worked, but would it have really been worth it to management to reduce the probability of this test failing? Like I said, maybe they aren't being careful enough, but I don't think we have any real evidence for that right now and I personally doubt it.

I think this is where NASA went wrong in many respects. They got so risk adverse, they built a bureaucracy of checks and oversight that moves slow. That’s one of thew reasons the industry needs fresh blood like SpaceX

Two reasons and probably more:

1. The risk is different when there are human lives on the line.

2. NASA is publicly funded and if they want to keep receiving funding, they can’t be blowing up the public’s money.

I feel NASA's approach is right for manned spaceflight, but is not sensible for robotic missions - metal is cheap, design time is expensive. I feel the public appearance is not a hard - nosed calculation, but Is instead a pr requirement

NASA has different risk categories for manned spaceflight and robotic missions, depending on cost.

I actually disagree with you. For one, consider that Starship is ultimately intended for manned spaceflight... yet SpaceX is still trying things and blowing stuff up (or imploding or...)

From my armchair perspective I think the risk adversity criticism is justified. NASA should take much more risk even in human spaceflight... in the early stages of platform development. First see if you can build the damn thing and build it repeatedly. Later, after that period, reduce risk and prove out and refine things to the human spaceflight standard. If you jump immediately to, "my God people are suppose to fly on that thing!" well, you're probably going to pay the Russians a lot of money for a fairly long time.

I do agree with you that the political management becomes a real issue. Politicians and bureaucrats are naturally disposed to a certain cravenness about such issues once they get to significant positions of power (senior execs of many companies do, too). But I think a charismatic leader that's out in front of the messaging could pull it off.

If they actually test and launch the damn thing, we would be further along with a safer rocket and don't have to pay the Russians.

Well, ironically and counter to your point, Russians had the same basic design now for nearly 60 years!

When a robot mission is wildly successful, they should build more of the design and launch another.

I think this is a very good point that is frequently overlooked, it would only cost 2x to have 10 rovers instead of one!

It is also absolutely embarrassing that we still haven't fully surveyed the moon for resources, dont know where all the ice is, etc. Lunar rovers should be numbering in the hundreds, maybe thousands.

Many of NASAs robotic missions are done through JPL which doesn’t follow all the same rules, but you’re absolutely right about the PR

Well because of that approach NASA lost the ability to do human space flight.

They literally couldn't do it, and the admired that they basically can't develop it internally without it costing 10s of billions.

So that's why the used the Commercial Crew program and that worked out for them.

The SLS is a good example, 10+ years of building at a cost far above 20+ billion until the first launch.

Prototype can and should be tested with the risk of blowing up if they want to improve rockets' reliability.

Obviously, there's not much tolerance when human lives on the line. But you don't have to test with humans.

They went too far and aren’t blowing anything up. There has to be an optimal amount of resource burn

Frankly I'd rather them blow up my money than spend twenty times the original budget and ten times the schedule filling the pockets of companies like Boeing.

1. Keep in mind NASA has multiple Centers and only one (Houston) is dedicated to human-rated spaceflight

2. Dirty little secret: even when NASA uses “startups” like SpaceX to launch and things go awry, the taxpayers are still on the hook for the majority of the loss because the Agency is self-insured

They got really risk adverse and their vehicles aren't that reliable and safe at the end anyway.

A JPL project manager teaching a course I took blamed it on the public's "mathematical illiteracy." He explained that there are diminishing returns on mission success rates so NASA often had the choice of paying X per mission attempt each with an 80% success rate or 2X for a single launch with 95% success rate. 80% sounds bad but the chance of two launches failing is 4%, which means a 96% chance overall success rate. If the first launch succeeds, they can usually save between 20-40% of the total budget (the remaining amount is overhead to build backups and keep scientists and engineers employed in case the first launch fails). However, since NASA is no longer doing world shattering stuff like landing on the moon, the human echo chamber amplifies each failure and drowns out the successes. Since NASA has to justify many appropriations on a per project basis, failures lead to projects getting canceled and NASA doing less actual science, even though their budget rarely shrinks - with more and more money instead allocated to inefficiencies like maintaining unused real estate/hardware and spreading out manufacturing across all districts in order to curry favor to get more projects going.

That's just for robotic/satellite missions. It's ten times worse when a manned mission ends fatally.

> since NASA is no longer doing world shattering stuff like landing on the moon

NASA has its Artemis program which is doing exactly this, returning to the Moon with a goal of a longer-term lunar base.

And they just held a period of application (it closed a few days ago) for the next generation of Artemis astronauts.

Artemis : https://www.nasa.gov/specials/artemis/

Yeah they said they will return to the Moon or Mars (depending on the president) since Bush 1. That is almost 40 years.

So another new moon plan. Another Shuttle derived launcher.

Not surprising that nobody is getting very exited with Artemis.

The SLS and Orion can only be described as dead-weights on NASA since they are so incredibly expensive. SLS has been in development for over 10 years at more then 2 billion per year and it will not launch until 2022 hopefully.

> NASA has its Artemis program which is doing exactly this

They've been designing Orion for a couple decades now, and the SLS is a shuttle-derived expendable launcher where what we see as key innovations are mounting 25% more SSMEs (still expendable) on the bottom of the fuel tank, extending the SRBs by a couple segments and putting the payload on top of the main tank.

Ares was interesting and we learned, with Ares I, that we shouldn't put humans on top of an SRB. There would probably be cheaper ways to find that out, but at least they tried something different.

> 80% sounds bad but the chance of two launches failing is 4%, which means a 96% chance overall success rate.

I don't disagree but I also don't think it's always as straightforward as probability theory would make us believe. One can imagine situations where a certain amount of QA leads to a certain semi-predictable amount of failures that allows for calculated risks to be taken, but one can also imagine situations where paying 2X leads to better solutions and processes and not paying 2X leads to failure on both the first and second launches.

> 80% sounds bad but the chance of two launches failing is 4%, which means a 96% chance overall success rate

I'm sorry, but this argument is also mathematically illiterate. Probability only works this way if the events are uncorrelated, but it should be pretty clear they're not.

If the first rocket has a 20% chance to fail it is likely that the second one will also fail due to similar negligence.

It goes both ways. Perhaps they make the same mistakes, perhaps they'd learn from their mistakes, and would _not_ make the same ones?

You're right in theory but demonstrably wrong in reality. Just look at the failure rate of launch platforms in the last 30 years. If I remember correctly, launch failures accounted for less than half of mission failures and a significant number of those weren't explosions but burn rate problems that ended up in the wrong orbit.

Rockets aren't mass manufactured and their fabrication is barely automated. Nobody except SpaceX keeps their tooling online long enough to have correlating failures except for actual design issues which are fixed in later launches. The correlated failure comes from the actual mission hardware but launch schedules are so spread out that a second mission has plenty of time to fix issues discovered in the first attempt.

If rockets, which are made one at a time and launch maybe a few times a year, had correlating failures like you said, no insurance company would cover launches.

It's unclear to me what you are basing this on. How do you know that the "how careful do we want to be to avoid testing failures" parameter was exactly where SpaceX wanted it to be in this experiment? Or was at an optimal point for SpaceX in general? If you don't know this - then your reply doesn't really mean anything, and can be be used to justify any mistake whatsoever.

I definitely don't know that the parameter is where SpaceX wants it to be, or even more importantly where SpaceX should want it to be given the information available to SpaceX. That's why there is room for disagreement.

I do have some hints that it's probably more or less where SpaceX wants it to be, some hints that they have chosen this parameter rationally, and some faith that they aren't stupid. For instance, Elon often discusses future tests failing as acceptable, they've declined to put in place more stringent QA efforts, they're still mostly building the hardware exposed to the elements, Elon has discussed that currently their approach is building a lot of starships to iterate on the manufacturing methods and to help them make an easy to manufacture design - he has signaled fairly clearly via his tweets that actually flying the "ships" is secondary, and so on.

My prior is that the parameter is as likely to be set as too high as too low. The testing failure at this stage in development is weak evidence that it is too low, the rest of the information I've just talked about is more convincing evidence that it is an appropriate value or too high, hence I think the post I responded to is "probably wrong".

An example of where I think the level of caution was probably too low (or there were other issues, but let's pretend for a second that this is the only knob management has, as I said it is an over simplification) - the recent CST 100 orbital flight test failure. That test was intended to demonstrate they were ready to fly humans, instead they suffered a series of serious failures, failing that sort of test is pretty strong evidence that they weren't careful enough. They've indicated this type of failure was unexpected and that it will cost the company slightly under half a billion dollars, which is also pretty strong evidence that they weren't careful enough. Moreover I know that they neglected to use procedures such as end-to-end software simulation that would have reduced this risk, and that they were in fact contractually required to perform. Meanwhile I don't have any real evidence that they were careful enough with regards to that test. So starting with the same prior as above, I quickly fall to the "they probably weren't careful enough side".

> Or was at an optimal point for SpaceX in general

The optimal point is definitely not a global value for all of SpaceX. Human spaceflight programs have obviously set it very high instead of very low. As they progress from experimenting with manufacturing to flying payloads it obviously gets turned up. Etc.

The value of knowledge they obtain from a failure far outweighs the “cheap tests” as you put it.

This does feel like Musk's software background shining through:

* Fail Fast. * Red. Green. Refactor.

This is more of his automative background shining through. That's where the whole machine building the machine idea comes from.

Reply on news.ycombinator.com