Anomaly Updates | Hacker News Reader

Anomaly Updates

spacex.com

119 pointsyread81 comments

3 helium tank issues in 3 years

2014 https://en.wikipedia.org/wiki/Falcon_9_Flight_10 (delayed by SpaceX due to a first stage helium leak)

2015 https://en.wikipedia.org/wiki/SpaceX_CRS-7 (failure of a strut which secured a high-pressure helium bottle inside the second stage's liquid oxygen tank)

2016 (a large breach in the cryogenic helium system of the second stage liquid oxygen tank took place)

I note that the helium tank is inside the oxygen tank. I'm sure there is a reason, but there have got to be some major downsides.

It's probably to help keep the helium cold. Since you already have to maintain a large reservoir of a cryogenic fluid, why not make it do a little extra work to cool a substance with a lower boiling point.

The helium is used to keep the pressure on the oxygen tank constant once the tank starts to empty (basically the helium is used to push the oxygen down).

It's probably inside for mechanical reasons pumping liquid helium is a nightmare it sips through pretty much everything (it can even go through glass when sufficiently cooled), in this case you don't really need to worry about all that stuff since you simply let the helium out and the difference in density would sort everything.

As for the reason it's not used to "cool the helium" it's because helium boils at -268.9c the liquid oxygen is cooled to -182.9c or to ~-206c in cryogenic mode, this is still way above the boiling temperature of helium. This would be about as effective at cooling, as cooling boiling oil with boiling water, incidentally it's also about as explosive ;)

No... I took too long to write my comment.

A temperature difference of 63 °C is still an awful lot easier to deal with than a temperature difference of 280 °C

That isn't how physics work, when something boils it boils. Also when you factor in heat transfer coefficients then the liquid oxygen in tank is actually "heating" the liquid helium pretty substantially as it has considerably higher heat conductivity than air. ;) The analogy of boiling water into boiling oil is pretty close one as far as the process goes, the boiling oil would evaporate the boiling water instantly and things go boom, the liquid oxygen would evaporate the liquid helium instantly and things go boom.

The helium is in the gas phase and at thermal equilibrium with the surrounding oxygen.

And 757 times it's liquid volume, hence the boom part.

The helium tank is inside the oxygen tank to keep it cold. For a given pressure and tank size, more helium is stored in the tank at lower temperatures.

For an ideal gas, PV = NkT, where P is pressure, V is volume, N is the number of atoms, T is temperature, and k is a constant. It looks like the O2 temperature for this version of Falcon is 66 K. If we take T_out ~ 273 K just for the sake of the discussion,

N_in / N_out = T_out / T_in ~ 4.

It's a significant difference.

That's not correct notation for the ideal gas law.

Ideal gas law doesn't apply to real gasses, (actually in some cases especially helium).

Ideal gas law does not apply to liquids.

Ideal gas law only works at low pressure and high temperatures.

Overall the ideal gas law is not a law it's an approximation that works in a few cases, it has no baring in this case.

You are right that the ideal gas law doesn't describe real gases but it can be a reasonable approximation for some applications, even at fairly low temperatures, provided the density isn't high.

Helium is the closest thing we have to and "ideal gas" as far as the ideal gas law goes when it's at STP.

It can still somewhat behave as an ideal gas as low as 50K or so granted it is not under pressure.

Once you increase the pressure the ideal gas law breaks down pretty much completely, and is not applicable to liquid helium at about 6000PSI.

People also forget that the tank is made out usually titanium or aluminum (i think aluminum can't hold helium but am not sure about that) wrapped in a composite material usually Kevlar/Epoxy or some sort of carbon fiber composite, extreme low temperatures are well a bit tricky for these materials.

NASA had issues with that during the shuttle program, and it seems that even the new Orion has encountered issues with the effects of low temperature on COPV's which required them to develop new resins and composites http://www.compositesworld.com/articles/orion-composite-tank...

FWIW: I am not a rocket scientist.

I'd be willing to bet it is to reduce the efficiency loss due to vacuum. The fuel tank (oxygen tank) is going to empty itself. However, you can't have a vacuum in the tank. So do you acknowledge that the rate of flow of fuel exiting the tank will be slowing down over time or do you keep the tank "full" to ensure constant pressure?

Helium, is inert, less dense than oxygen and cheap. It's probably the best space filling collection of atoms, and simultaneously it's density[1] forces the oxygen further "down"[0] the fuel tank.

[0] down is some direction relative to thrust. [1] density according to: "weight" generated by thrust.

I'm addition to thermal issues others have noted...

One reason is to keep the pipes as short as possible.

Another reason is for internal volume efficiency. You could put the helium tank and other support equipment outside the LOX tank. However, there will naturally be some void spaces around all this stuff. This results in the second stage being larger and heavier. Putting it all inside the tank means that all the nooks and crannies can be filled with LOX.

So these high pressure tanks must be an incredibly hard/complex problem?

Alternatively they're not incredibly hard / complex so not enough attention is being given to testing, manufacturing, etc. Though I'm curious which way it is.

They're a complicated problem, but I can't help but wonder if SpaceX underestimated the difficulty. They used to have a 3rd patty supplier for the helium tanks but they decided to manufacture them in-house a few years ago. As far as I know, all 3 tank problems have happened since they started producing their own.

The strut which failed in 2015 was outsourced. A tiny fraction of the struts from the factory failed at well under the specificed maximum load, and one of them ended up in that rocket.

That or spacex overlooked the complexity

You're describing 3 extremely different issues though -- it's easy for something going wrong to involve helium tanks because it's something that's used in the rocket that can lead to an overpressure event.

At this stage of the investigation, preliminary review of the data and debris suggests that a large breach in the cryogenic helium system of the second stage liquid oxygen tank took place.

...

Pending the results of the investigation, we anticipate returning to flight as early as the November timeframe.

Note that the second-stage helium system was also the source of the previous explosion of a SpaceX mission to the International Space Station (which blew up during launch).

https://en.wikipedia.org/wiki/SpaceX_CRS-7#Launch_failure

But:

> Through the fault tree and data review process, we have exonerated any connection with last year’s CRS-7 mishap.

I found that puzzling too... doesn't this mean it was the same component, though perhaps failing in a different way?

Sounds like that pressurization system needs to be taken out back and shot.

Frankly, that's how you increase your failure rate. If a system fails, you figure out why, figure out any other failure cases that may have been missed, and you fix them.

You don't ditch something that largely works, and replace it with something brand new, for which you have zero data. That's how you lose rockets, cargo, and lives.

Same thing applies to software: you encounter an error, you analyze why it happened. Then ask: what other places could have this error? Can I eliminate this entire class of error by design or other method or do I need to fix them all individually?

You organically end up with a more robust system that way. Don't throw out the baby with the bathwater.

And then there is the "Did you turn it off and on again" school of thought. I've had to fight against that so hard before when the correct issue is inducing something to drop core, or fix a leaky program using valgrind.

Assuming they were correct about the first incident, it was caused by a faulty strut that was supposed to hold the helium COPV in place, but failed. If they've already ruled out a strut failure in this case, it seems pretty safe to say that they are unrelated.

Though it must be said that NASA apparently never signed off on the strut explanation. So conceivably it was never the strut, in which case it could be the same issue. Then again, I assume the people working on this are much smarter than me and have much more information, so they probably considered and rejected that too.

The same system. There would be multiple components of the helium pressurization system that have failure modes causing tank overpressure.

The previous failure was with a strut holding the tank. That's not really the same 'component', depending on one's specific definition.

Perhaps taken out back and blown up would be more apropos?

Only in a loose sense. The CRS-7 failure was a strut failing which held the tank. This time, the tank itself is suspected to have failed. These are probably completely unrelated.

They might turn out to be related in an unexpected way. For example, they might have redesigned the strut in such a way as to cause excessive stress on the surface of the tank near the attachment point, leading to rupture.

These systems are probably more complex than the cruddiest Enterprise software that any poor soul has ever had to maintain, so I guess anything is possible.

George Carlin would have a field day with that wording.

I love/hate the carefully curated language.

"Three weeks ago, SpaceX experienced an anomaly at our Launch Complex 40 (LC-40) at Cape Canaveral Air Force Station. This resulted in the loss of one of our Falcon 9 rockets and its payload."

"Three weeks ago, SpaceX experienced a catastrophic failure at our Launch Complex 40 (LC-40) at Cape Canaveral Air Force Station. This resulted in a massive explosion/fireball that completely destroyed the rocket and its payload."

Personally I love the euphemism they use when they blow up: "rapid unscheduled disassembly"

No word of a lie, I do that all the time in Kerbal Space Program. I didn't crash, the mission terminated due to unscheduled lithobraking. My rocket design didn't explode, it had a excessive surplus oxidisation.

Precision, people. Precision with language is the mark of a clear mind.

<snark>

I think i have seen NASA use that one before.

Reminds me of the first episode of the west wing, after President Bartlett apparently rode his bike into a tree.

"What do you want me to tell the press? That the president, while riding his bike, came to a sudden arboreal halt?"

RUD is at least used ironically. I don't get the impression that "anomaly" has the same tone.

"just had dinner with a credible source i trust that spacex is about 99% sure a COPV issue was the cause. 'explosion' originated in the LOX tank COPV container that had some weird harmonics while loading LOX."

(COPV = Composite Overwrapped Pressure Vessel, they are titanium bottles wrapped in layers of continuously wound carbon fiber + resin)

https://www.reddit.com/r/spacex/comments/545dju/unconfirmed_...

I wonder if a general policy of using randomly fluctuating flow curves instead of flat ones would decrease the likelihood of harmonics.

Presumably, loss of the cryogenic helium through a "large breach" would lead to rapid warming of the liquid oxygen and equally rapid over-pressurization of the tank. So this is consistent with the source's contention that the explosion originated in the LOX tank, although the COPV may not be the root cause in that case, depending on the over-pressure it was rated for.

Would it be the LOX or the HE rapidly warming? (I thought helium would have the lower starting temperature ...)

The helium isn't "cryogenic" the LOX is, Oxygen is liquid at 182.9c, SpaceX cools it down to ~-206c when they load their fuel in cryogenic mode.

Helium boils at -268.9c, basically when liquid helium meets liquid oxygen it boils and expands to 757 times it's volume, which means boom.

Yes, I misinterpreted the article. Here's a clearer description from spaceflightnow.com (https://spaceflightnow.com/2016/09/23/falcon-9-rocket-explos...):

"The Falcon 9 rocket’s upper stage liquid oxygen tank contains several composite helium vessels, each pressurized to about 5,500 pounds per square inch in flight. The helium is routed through the second stage’s Merlin engine, where the helium warms up and is injected into the rocket’s propellant tanks to pressurize the stage as the launcher burns fuel, keeping the tanks structurally sound."

The most interesting hypothesis I've heard is intentional sabotage, like with a sniper rifle shot. Snipers have been known to hit human targets from 1.5 miles away. It seems like a huge rocket can get shot from a much larger distance.

It's possible, and there could be motive from competing companies, but frankly it seems a bit far-fetched.

Not just companies, countries like Russia. SpaceX is eating their space lunch.

There's also the whole conspiracy theory with Israel not wanting the launch to go through because it was a condition for the sale of the Israeli Spacecom to the Chinese, and Israel wanted to maintain control of their space industry.

The Israeli government approved the sale, they could've said no if they wanted too.

If some one doesn't want Israel to transfer satellite technology to China it's not really the Israeli government.

Also Spacecom is just a satellite operator, the satellite was built by IAI and other contractors, IAI is a state owned corporation which builds Israel's satellites and space launch vehicles.

For the most part Spacecom doesn't have anything of real worth the transfer (other than their commercial operations), they do not operate Israel's military satellites they do not even operate Israel's commercial dual-use earth observing satellites.

Even more far fetched, someone shorting a large amount of SpaceX stock. That would make for a decent movie plot though.

SpaceX isn't publically traded IIRC, so shorting the stock isn't really possible.

How big is the upper-stage cryogenic helium tank? I imagine not very large, and it's not obvious to me exactly where it is placed. The hypothesized saboteur would have to be an amazing shot!

The volume of a rocket is predominantly composed of fuel. And even if a puncture doesn't cause an anomaly at sea level, it most certainly will upon some flight condition.

You don't shoot the rocket in a static fire I'd you want it to explode in flight, though.

They don't have to hit just the helium tank to cause the explosion. Any fuel tank would probably do.

If the failure has been isolated to the helium tank, that further debunks the 'sniper' hypothesis, right?

In what way?

Well, if any tank failure would cause an explosion, wouldn't the hypothetical sniper aim at a big tank? Not a tiny flask located inside another tank.

The hypothetical sniper could have been aiming at a large tank, missed, and luckily hit the helium tank. Hypothetically, of course. :P

If the smaller tank is contained within the larger tank, that seems like an impossibly lucky shot.

They're not that large, about the size of a person;

https://i.imgur.com/K3H7mdY.jpg

A sniper could hit that from 1 km distance.

I'm not saying that's what necessarily happened. I'm saying a sniper could very well do that.

Well with the rifle there is a problem that bullet would be left on the stage. Not sure whether would it be possible to do it with for example laser.

How is that a problem? What about the giant debris field left?

With an explosion that large, you would never find the bullet.

Alright, everybody, start looking for the grassy knoll...

What is it with HN and belief in various absurd conspiracy theories? Is it an American thing?

It's a human thing

Nit: the contention that the world's leaders are shape shifting alien reptiles is an absurd conspiracy theory. This one is just extremely unlikely in the absence of direct evidence.

Musk is the new Jobs on HN. His company is beyond failure for he is a holy figure to be worshipped and everything his company does is not the result of the employees, but of his blessing and ability.

Interestingly, I have noticed a tendency for positive stories and statements about SpaceX (and, if I wanted to be contentious, I could even call them 'fanboy' originated) to refer to him as 'Elon', whereas negative or neutral commentary uses the less familiar 'Musk' instead. I'm not quite sure what conclusion to draw from this, though. As for the 'ElongatedMuskrat' user on Reddit, I have no idea...

You're confusing a hypothesis and a belief.

I think there is a big cultural element. Not to say other cultures are immune to conspiracy-theory thinking, but something about American exceptionalism seems to encourage it.

Aren't religions (at least Abrahamic ones) just one big conspiracy theory? People believing in some kind of series of events with little to no evidence? If so I think people globally are apt to believe in dubious claims, aka conspiracy theories.

That could certainly be done, but I don't think it could be done in a way that wasn't obvious to investigators.

You do realize they are launching from a military base right?

Reply on news.ycombinator.com