SpaceX CRS-7 Failure Investigation Teleconference Thread
reddit.com
reddit.com
Elon at 36:15
Two thoughts: if you need to send a message like that, the culture of your organization needs some work. Also, does he really think a person can send a message like that - when their manager does not agree - without torpedoing their career?
Actually it's only a problem if you're REALLY wrong. If you're just a little bit wrong, or your manager was only just barely right, it's likely that your manager will be reprimanded instead of you.
Further I think half the point of the exercise is to help ensure that managers don't dismiss people's concerns. If you want everyone to take launch safety really seriously broadcasting to everyone that you can go above as many boss' heads as necessary to get something addressed should help keep everyone on their toes.
It is worth studying how that kind of thing has NOT fixed the issues as NASA, and how it did not prevent us from losing Columbia. You either have a culture that puts safety, engineering and problem solving ahead of politics and expediency in all cases, or you don't. There are absolutely not any shortcuts to that.
Besides, this is unusual for a company the size of SpaceX. In most companies of some size I've worked at, most people may have the CEO's e-mail. Maybe. But usually the one filtered by a PA. But most of them don't have the CEO's cellphone number.
> Also, does he really think a person can send a message like that - when their manager does not agree - without torpedoing their career?
If this is usually the case, does that not mean that all organizations needs to work on their culture? So what is the issue with sending am message like that again?
At Nasa, people raising red flags before the launch of a manned vehicle? STS-51-L, Challenger, January 1986
I just listened to a Freakanomics podcast about the person who raised a red flag and wanted to cancel the launch. Very interesting listen. Episode was "Failure Is Your Friend".
http://freakonomics.com/2014/06/05/failure-is-your-friend-a-...
The liquid oxygen tank provides the liquid oxygen which fuels the rocket. They use helium to keep the tank pressurised as the liquid oxygen gets used up (why helium? it's light and unreactive). The helium is stored in its own tanks, which are secured by struts like in this picture: http://i.szoter.com/741dc2bcf5762a48.jpg (via reddit). The strut failed, causing a helium leak and a complicated series of events (including the helium tank bouncing around the liquid oxygen tank?), that basically resulted in the liquid oxygen tank being over-pressurised and exploding.
Bonus (very cool!) gif of the liquid oxygen tank during a previous launch (unfortunately cameras apparently weren't included in this launch): https://i.imgur.com/WRp2ujX.gif (also via reddit)
Some extra cool things:
(1) They narrowed down the source of the failure using "acoustic triangulation", which, I think, is essentially using sound sensors (accelerometers?) located at various locations to pinpoint the location of the failure in 3D space.
(2) The Dragon capsule could have been saved if it'd had the right software (which would deploy the parachutes). They'd already planned to do this, and will now have a software update for the next launch. Why hadn't they done this already? Because if the parachutes deploy accidentally, it could result in launch failure, so it's something they have to be careful about. But the capsule survived the explosion and they remained in contact with it until it was below the horizon.
My understanding is that they have them on every launch, but bandwidth constraints with the downlink mean they have to pick between the many cameras to monitor in realtime.
It's also one of the few gases that will stay at pressure even at the temperatures of liquid oxygen.
Other choices are neon (but it's heavier) and hydrogen, which would react with the oxygen.
An accelerometer is a poor device for doing acoustic measurement. At best, it can measure how the surface the accel is attached to responds to the acoustic environment (and that's only if the accel has a very high frequency response and is conditioned and sampled appropriately). For this kind of work they probably wanted to measure dynamic pressures within the tank, which they would most likely do with microphones. Other than that you have exactly the right idea.
Of course, a microphone was designed with high bandwidth in mind, while accelerometers probably value more precision around the DC input.
Your point remains though, if the accel has appropriate frequency response and is properly conditioned and sampled at a high enough rate, acoustic measurement is possible. What I was trying to say (maybe poorly) is that an accelerometer would not be my first choice for doing such work. But in the aftermath of an event like this, the data may be (and was, it sounds like) usable for such purposes.
> He said the early team had "an extreme level of paranoia" because of the difficulty of learning how to design and launch rockets. But now, “the vast amount of people at the company today have only ever seen success… when you’ve only seen success, you don’t fear failure quite as much."
> Musk said the night before each launch, he sends a company-wide email asking employees to send him an email or call his cell phone if they have any reason to believe the rocket should not launch.
I think it's good such realities are being realized, then handled in a gentle (but firm) way.
Source: http://mashable.com/2015/07/20/spacex-launch-failure-strut/
The problem might very well be that the very tests a part must pass will weaken the part to the point that it is not usable or less reliable. In other words, some tests are destructive.
Here's on of many interesting pages that came up by searching for "how to test aluminum for grain structure".
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
I guess I am saying I hope a vendor isn't blamed for this when the reality of the matter might very well be that testing to 100% certainty is impossible.
Edit: i.e. NDT'd pieces are not immune from failure (but other factors such as age and in-service time/rotations/re-thread/re-collar runs etc. can provide additional context to NDT results and should correlate more or less with actual failure patterns).
In a single-use part, non-destructive testing mightn't be a complete picture of how the part will perform.
A part being mildly out of spec might not be a vendor problem. A part being 20% strength is absolutely a vendor problem.
Why was this designed such that the failure of A SINGLE STRUT would be catastrophic?
I don't like to blame a vendor for what should be an engineering problem, whether this means design or testing.
The problem here --assuming it is as described-- is that someone designed a system with the assumption that none of these struts would fail. And, furthermore, executing on a design where the failure of ONE strut could cause a disaster.
Anyhow, that's what it looks like to me given what's been released.
I could be totally wrong.
There are plenty of things in a rocket that can fail that would take the entire vehicle with it. Structural mechanics is pretty well understood and loads are well predictable, so it seems perfectly reasonable to me to design with the assumption that a part with a 10x safety factor will not fail.
If you didn't, your rocket would never get to orbit anyway.
I suspect SpaceX's costs are going to, over time, increase significantly as they continue to learn that playing it loose isn't always possible in that business. They are famous for going for COTS in order to save money.
This is NOT a put-down. I think what they are doing is fantastic. It is obviously redefining aerospace. At the same time I am astounded that critical structural components are not 100% tested. That said, metalurgy isn't my area of expertise, which means my opinion here could be complete nonsense. Perhaps this particular failure mode can only be tested through destructive methods (sectioning?) which means you can never be 100% certain to be flying good metal.
I'd argue that the likely result here is Elon Musk will continue his campaign of "hell with it, we'll build our own".
On the contrary, SpaceX is famous for vertical integration. COTS may have been the idea back in the Falcon 1 days, but those days are long gone.
Being a fan and all is admirable, but I don't think it's "sad" that the Internet wasn't given a seat on the call. SpaceX has a business to run.
As far as I can tell, they write quite a lot of their CA* software ;).
snippet:
"Preliminary conclusion is that a COPV (helium container) strut in the CRS-7 second stage failed at 3.2Gs. A lot of data was analysed, it took only 0.893 seconds between first sign of trouble and end of data. Preliminary failure arose from a strut in the second stage liquid oxygen tanks that was holding down one composite helium bottle used to pressurize the stage. High pressure helium bottles are pressurized at 5500 psi, stored inside in LOX tank. Several helium bottles in upper stage. At ~3.2 g, one of those struts snapped and broke free inside the tank. Buoyancy increases in accordance with G-load. Released lots of helium into LOX tank. Data shows a drop in the helium pressure, then a rise in the helium pressure system. Quite confusing. As helium bottle broke free and pinched off manifold, restored the pressure but released enough helium to cause the LOX tank to fail. It was a really odd failure mode."
The delays will be in setting up a QA process to test each individual strut part, as well as eventually testing all vendor-supplied parts. If anyone can do it, SpaceX can. No reason they can't design robot systems to test every part.
If your stuff is mission critical, if anybody could potentially die, you really can't trust the parts from these vendors. A few of their smarter customers would repeat all of our tests and send back defective parts. The thing was, we had a lot of borderline stuff come through from bad production (poorly paid or poorly trained staff or defective tools or materials), and once some of these parts hit QA, they had a lot of expense sunk into them. The company didn't want to eat that cost, so there were a lot of arguments between myself and the general manager. I failed a lot of stuff that previous people in my position had let slide.
They also had a really stupid hockey-stick output graph each month. The beginning of the month was slow, we were all cleaning our work areas and retesting our test equipment, and then the last week of the month they'd try to produce 90% of their expected output for the month. Because of my reputation for rejecting stuff, he'd hover over my work area for the last day or two each month.
Given the size of the company I worked for, I have to assume this is not uncommon practice.
It was a heck of an experience, I finally got a better understanding for why so many things seem to break all the time.
Kudos to you!
You really nailed it. The GM's position -- and he said this more than a few times -- was that the parts were designed with extra tolerances already, so if they were a little below spec it was OK.
Engineers have to keep that in mind when designing products: production knows there's a margin for error and they'll take that into consideration when deciding whether or not they can get away with shipping something.
(And the GM was a pretty OK guy, we got along fine otherwise. He in turn was just under a lot of pressure from further up the ladder to meet certain production goals.)
“A laser weld, sir.”
“So it would appear. Your identification is quite understandable --- and quite wrong. I want you all to memorize this piece of work. Look well. Because it may easily be the most evil object you will ever encounter.”
They looked wildly impressed, but totally bewildered. He commanded their absolute silence and utmost attention.
“That,” he pointed for emphasis, his voice growing heavy with scorn, “is a falsified inspection record. Worse, it’s one of a series. A certain subcontractor... found its profit margin endangered by a high volume of its work being rejected... The welds passed the computer certification all right --- because it was the same damn good weld, replicated over and over again...”
He gathered his breath. “This is the most important thing I will ever say to you. The human mind is the ultimate testing device... There is nothing, nothing, nothing more important to me in the men and women I train than their absolute personal integrity. Whether you function as welders or inspectors, the laws of physics are implacable lie detectors. You may fool men. You will never fool the metal.”
--- Lois McMaster Bujold, _Falling Free_
[0] https://en.wikipedia.org/wiki/Statistical_process_control
The funny thing was Delphi was also a customer (the electronics subsidiary of General Motors). They wouldn't pay for more than normal statistical sampling. When he visited them, they had large bins of defective car radios that wouldn't turn-on, etc. To Delphi it was cheaper to run the production line flat-out and deal with the failures, rather than find problems earlier by inspection of received parts.
http://www.thisamericanlife.org/radio-archives/episode/561/n...
Can they actually do that? Aren't most of those tests destructive?
"And thus it was learnt that on the twentieth repetition, company-wide emails cease their function, having been seen and imprinted many times, as it was with the boy who cried wolf. But with the experience of failure comes wisdom and strength for the future of the civilization who wishes to become spacefaring." Elon 36:15
The way half the people on this site worship Musk is freaking creepy. Just let it go, Daddy Elon doesn't love you.
Although, I'm fine with churches keeping their tax deductions if I can get deductions for donating to SpaceX or Tesla.
Meanwhile everything they do or use or enjoy is based on the work of the real world...
I'd also note that 'tribalism' is a fundamental human charactistic. Of course we tend to gravitate toward, socialize with, and give preference too those we perceive to be like us. Whether it's people who cheer for the same sportsing team, the same ethnic group, the same school, the same political party, it's all really isomorphic to those who share a religion. So I also find this idea that people who believe in magical worlds have some sort of advantage over those that don't, to be similarly without basis.
But perhaps I missed your point and you meant something else?