Hacker News Reader

Comment by brianshaler | Hacker News Reader

It's surprising how much of a disruption this caused to their launch schedule, but it's great that they got to the bottom of the issue before repeating the process on any further rockets.

The worst kind of bug is the intermittent one that you can't reproduce or see in logs. With only 35-55 milliseconds of metrics leading up to the event and nearly obliterated evidence, it seemed nearly impossible they would be able to nail down the exact cause.

Critics have pointed out risks in SpaceX's method of loading supercooled fuel. Accidentally forming solid oxygen around carbon fibers from a buckling overwrap is an example of this.

There are benefits, but they will need to find and prevent all these possible edge cases exposed by the new method.

View on HN

This "anomaly" caused hundreds of millions of dollars in losses, and was an unprecedented failure mode. It doesn't surprise me at all that it disrupted the launch schedule for the better part of a year (edit: actually the lesser part of a year, about four months total if they launch soon). If they manage to return to flight this month, it strikes me as rather quick.

I wonder if supercooled propellants are really worth it. It seems like a huge risk in exchange for a small performance increase. Hopefully this will be the last vehicle loss they suffer from it.

"small performance increase"

FT can get 30% more mass to LEO. That is a pretty big performance increase, especially considering that it enables SpaceX to carry the maximum payload of v1.1 in expendable configuration and still land the FT rocket. Even if we assume that half of that performance boost came from the other improvements, 15% is a significant increase.

That's the seductive side of design tradeoffs with rockets.

Shaving just a little weight (especially on the 2nd stage) or squeezing out just a little more performance can have dramatic differences in mass to LEO.

The question is how much this will increase overall operational cost. Up until this year or so, it has seemed that SpaceX has been willing to compromise on performance to get lower overall operational cost and reliability.

I'm a bit worried that they've started to go the other way with some of the improvements they've made this year.

It disrupted the schedule for 3 months, not for "the better part of a year". Otherwise you are making a fair point (but SpaceX seem to believe that it is worth it and that they will iteratively fix all these issues without losing customer trust).

Huh, I was sure it was in June. Actually September 1st. I guess it just feels longer to me. Thanks for pointing that out.

Following STS-51-L (i.e., the Space Shuttle flight on which Challenger exploded on ascent), the Space Shuttle fleet was grounded for a bit over two-and-a-half years. This is quick (OTOH, this is unmanned!).

After the Ariane 5 first-launch failure, it was well over a year before it flew again.

My understanding is it effectively reduces launch costs by ~25% by getting ~30% more mass to LEO. Really if it increases risks of failure by less than 10% it's probably worth it for unmanned rockets.

PS: High launch costs set up a cost escalation as satellites are so expencive they can't fail. Which means launches need a really high success rate. If you could get 10,000kg to LEO for say 1 million but had a 40% failure rate you would see a very different approach with much lower overall costs.

With this explosion, there was a ~$200 million satellite destroyed as part of a ~$60 million launch. We'll probably see much cheaper payloads to go along with cheaper launches eventually, but for right now the payload loss dominates, so you have to look at that as well.

For this particular problem, the risk to the payload will be eliminated by not attaching it for the static fire. Any risk during the actual launch would still affect the payload, though.

Payloads are insured, most likely SpaceX itself takes most of the loss.

The insurer took the loss on the satellite, SpaceX took the loss on the vehicle/launch contract, and the pad equipment.

Insurance isn't magic money. The cost of the satellite is recouped through premiums charged to companies launching satellites. It becomes part of the cost of launching one way or another. If gain X has cost Y, Y still factors into the tradeoff even if it's insured.

To put it in concrete terms, if insurers decide that SpaceX is significantly more dangerous to its payloads, then launch insurance for satellites using SpaceX rockets will become significantly more expensive.

Not sure what you're getting at with the condescending tone. My point is that the satellite owner did not realize a $200m loss; SpaceX being at least partially self-insured, from what I've read, is more likely to take a hit. There were 85 launches in 2016, one rocket blowing up is not going to significant change the landscape (heh).

Disclaimer: I don't know shit about the space industry, so don't waste your time.

We're discussing the tradeoffs involved in making launches cheaper but more risky. My point here is that insurance is irrelevant to that tradeoff. One way or another, the payload owners are ultimately paying for that risk.

This fault wasn't due to the introduction of supercooled LOX, though. They've been doing that for a while. The change here was to the tanking schedule so that it could better withstand a last-minute hold. As it stands, they're back to any hold during tanking leading to a launch scrub.

Once the redesigned tanks come in, it's likely that they'll go right back, because little holds aren't uncommon.

They'd been doing it for about nine months before this explosion. It wasn't the proximate cause, but it was a major change to the vehicle, one which is highly unusual in rocketry, and without that change, the proximate causes for the explosion wouldn't have happened. Without supercooled propellants, there wouldn't have been the need for faster tanking (because the fuel can just be at ambient temperature, and the LOX maintains temperature by boiling), and even if there were, the LOX wouldn't have been cold enough to solidify.

I find the whole idea of carbon fiber inside a pressurized LOX contain a bit risky. Also, what about all those reports that we're running out of helium? Is that credible threat to ongoing space flight?

We're not really running out, it's just getting more expensive because the government doesn't regulate it's availability anymore since the strategic helium reserve was done away with. It won't be a big deal for (already expensive) rocketry. Buying helium balloons for your birthday will be pricey, though.

Um, no. We are in fact running out of helium. There's only a certain mount and nobody not even natural processes is making anymore in any significant quantity. And what we do have is rapidly being released into the atmosphere and escaping into space.

And given that the nearest natural source of helium is Jupiter, that is a little concerning.

New helium is generated by alpha decay, and it comes up in natural gas wells that happen to be located in areas with natural radioactivity.

Not much, though. A quick perusal of Wikipedia (buyer beware, etc.) indicates that about 3,000 tons of helium are generated per year through radioactive decay (that's all happening in assorted rocks throughout the Earth's crust, so much of it wouldn't be easy to recover, either) while current production rates are around 9,000 tons per year.

A natural gas well has 10s of millions of years of helium accumulation from its local area.

Right, which means that while it's renewable over geological timescales, it's more or less a fixed reserve in human timescales. Much like oil.

Avogadro's number is very big: 6e22.

It's even bigger than that: ~ 6e23.

https://en.wikipedia.org/wiki/Avogadro_constant

Thank you.

los grandes avogadro's :)

You can extract helium from natural gas.

An estimated 54 billion cubic feet (1.5×109 m3) of helium was detected in Tanzania in 2016

https://en.wikipedia.org/wiki/Helium#Modern_extraction_and_d...

Yes that is where all helium on earth comes from: trapped ground sources, often in natural gas and oil deposits. Alpha particles from the Earth's radioactive core propagate upwards and get trapped somewhere in the earths crust, where we find them. But it took hundreds of millions of years to create macroscopic amounts of helium, so once all the sources in the ground or tapped it's gone, on human timescales at least. And you can't recycle from the atmosphere because it escapes the Earth's gravity.

I'm pretty sure we could make more helium if we really needed it. Nuclear fusion produces helium as a waste, and I expect fusion to become widespread before we run out of natural sources of helium on Earth.

We already rely on nuclear reactors to get several elements that are used in radiotherapy. IIRC there was a major shortage of medical isotopes a few years ago when a Canadian reactor that used to supply most of them was shut down.

Of course, it will be expensive, so no more cheap helium balloons for our grandkids. For specialized applications, though, we can afford some artificial helium.

A big difference between radionuclides and helium is that radionuclides are only needed in a relatively "small" quantity (producable by fission). Nuclear fusion would create helium but in very small quantities. Having 1TWe of fusion reactors would only yield 5grams of helium/second. That is not much.

> And given that the nearest natural source of helium is Jupiter, that is a little concerning.

Sun is closer and produces helium all the time, but less practical to harvest it from.

The US government alone is in the process of disposing[1] 1 billion cubic meters of the stuff from it's strategic reserves dating to back when airships where a thing. That volume is equivalent to a cube 1km long on each side.

1. https://en.wikipedia.org/wiki/National_Helium_Reserve

And when it is gone?

I believe that it's very likely that SpaceX can recover the helium used during operation. The helium isn't expelled from the fuel tanks. Since the 1st stage returns, it could be recovered.

Its more of we are running out of cheap helium as the US has been selling off its massive stockpile and its nearly depleted.

The cost of the physical helium in rocketry is going to remain negligible for a long time. The total cost of materials in a modern rocket is something like 1/2 of a percent of the cost of a launch, and I'm confident saying Helium is a tiny fraction of that. It doesn't matter if it goes up in price by a few orders of magnitudes. We are also still producing it in large quantities (natural gas often contains a lot of helium, which we extract). Rocket's also use a tiny amount of helium percentage wise, simply because there are so few rocket launches.

However even if we did somehow run out of Helium entirely, once SpaceX switches to Methane fuel (as they plan to do for the ITS) they won't need it anymore. Currently it's used to pressurize the tanks, but Methane does that for itself. It would mean speeding up development of 'small' methane launch vehicles presumably, but that's it.

I imagine that there are other ways to accomplish the same goal, as well, such as using other noble gasses. Helium provides the best cost/benefit tradeoff but surely other approaches would be practical if helium becomes unavailable.

How would you pressurize the liquid oxygen needed for the methane to burn?

Per the other comment, no other noble gas has a low enough boiling point to remain a gas at LOX temperatures...

If you need to you can heat up the oxygen to increase it's volume until the pressure is correct. I forget the reason that it's easier to do with methane than oxygen but it's possible to do with oxygen too if you really need to. ULA's new ACES upper stage is planned to use autogenous pressurization for both it's oxygen and hydrogen tanks.

How does the gas get into this void between liner and overwrap? I presume the liner is meant to be impermeable? Or is it only impermeable when properly backed with the overwrap?

The helium COPV is submerged in LO2, so it's the oxygen that's soaking into the wrapping. I think the failure is that the oxygen is soaking into the wrapping before the helium is fully pressurized, and then the metal container expands slightly and crushes the oxygen that soaked in between the metal vessel and the carbon wrapping.

To really be sure they got to the bottom of the issue and are not seeing an intermittent issue they should duplicate the failure, but it may not be possible to do this safely. I detect a little bit of hand waving around the source of ignition:

"breaking fibers or friction can ignite the oxygen in the overwrap"

I would like to see at least this ignition source duplicated.

I also found it a little surprising with so much involved that they didn't take the effort to reproduce the failure case, perhaps doing so on a smaller scale than a full vehicle launch if possible.

The problem is that accurately reproducing this failure case is very difficult. You need to replicate the thermal properties of the loading process and the tanks, at the same speed with the same amounts of propellant and helium, effectively meaning that you have to do it with a full-size tank, and with a large amount of pad equipment. This is all extremely dangerous and almost certainly will result, if successful, in the loss of the testing stand.

It doesn't specify whether the mechanism proposed to have caused this particular anomaly has been experimentally verified to be able to cause ignition but, even if it had, it would still not prove it was the mechanism that was at work in this particular instance.

Yes, proposing a theory and reproducing a result experimentally doesn't prove anything - but it is the basis of all science.

My point is that the wording "several credible causes for the COPV failure" does not imply they haven't experimentally verified that those causes can cause a COPV failure, which you seemed to imply.

Note that the company that used to produce COPVs for SpaceX had a large explosion during testing at their factory. Some testing now takes place on barges at sea, and the huge carbon-fiber tank for ITS was also tested at sea.

Who says they didn't repro the failure?

They have duplicated the failure at their test site, possibly a number of times. At least one was mentioned but there have been multiple explosions.

Reply on news.ycombinator.com