I found that under a simple Bayesian model, the estimated probability of catastrophic failure during the next launch is 3.6%. See http://bentilly.blogspot.com/2012/10/how-reliable-will-falco... for details.
Update: I made a careless calculation error. Change 3.6% to 2%. Oops.
Often a more useful and appropriate construct in the Bayesian world, is the use of a belief network or Bayesian Network. This is a probabilistic directed acyclic graph (DAG) that encodes priors, often in the form of subjective beliefs (yes subjectivity can be useful), including specific domain knowledge.
Common example: Consider a naive Bayesian classifier (a specialized form of belief network) that identifies individual pieces of spam. Do we arrive at the spam score by entering the probability of past events into a simple model based of the Bayes theorem formula?
No, it's trained using the vast amount of domain knowledge and pattern recognition (through our experience and own estimation of what 'spam' is) encoded in our minds, that provide the priors. Thus, even though there is a large amount of subjectivity involved, the overall result can objectively be measured, within a given utility function. Incidentally, this is often what makes many hardcore empiricists 'nervous', and hence avoid belief networks altogether.
Coming back to the Falcon 9: A piece of prior information outside the scope of historic safety records, for example, one of the lead engineers having a nagging doubt about a particular technical risk based on some observed phenomenon, could have an impact on the real world probability of the next event being a failure. (Which is a pretty useful thing to know!)
In fact, this exact scenario happened in 2003 with the disastrous destruction of the Space Shuttle Columbia. [1] An engineer spotted something wrong on previous flights, but management failed to heed the warning[2]. This could quite possibly have been averted, if a risk mitigation model were in place to account for such evidence.
Looking forward, it's quite possible to imagine a future where this decision making has been outsourced to a sophisticated AI based off a Bayes net, with far more accurate real world modeling of risk and failure probabilities, outclassing the amount of evidence and a human or committee could possibly hope to compete with.
While I've nothing against frequentist approaches (albeit Bayesian naturally makes more intuitive sense to me), a minor drawback is the reliance on the past to predict the future. For example if you had safety records on 1 million previous flights, then one might be tempted to say, "well that's that then, we now know objectively the probability of failures in the future -- end of story". But, the 1 000 001 flight may have been designed to fly on a completely different type of technology, that will change significantly change the safety record of space flight going forward for the next "x" years. Thus using a Bayesian approach account for all relevant priors, it would in theory be possible to reflect a more accurate probability for the 1 000 001 flight, before it took place.
Lastly Bayes nets are not the best tool for every job, and do have drawbacks in certain situations. They are vulnerable to things like Bayesian poisoning or confirmation bias. A Bayesian approach is only as useful as the ongoing real world relevancy and accuracy of the priors. As the old adage goes, GIGO - garbage in, garbage out.
[1] http://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaster
[2] http://www.guardian.co.uk/science/2003/jun/22/spaceexplorati...
You should know that there's no such thing as "real world probability". The rocket will crash, or it will not, period.[1] Probability, as a measure of your own ignorance, is subjective.[2] Your main point still stands though: knowing about the uncertainty of that lead engineer certainly should influence your assessments of the risks involved.
[1] What will actually happen is, the universe splits into many "worlds" (blobs of amplitude in configuration space), a fraction of which will have the rocket crash, and the rest won't. That's the closest thing we have from "real world probability", though it really isn't: the laws of physics as we currently know them are still deterministic.
Indeed. We pretty much agree then. If you re-read the "real world probability" in the context, I was talking specifically about a belief network. The degree to which a justified belief, in an outcome will occur. All beliefs are by their definitions 'subjective' and occurring in a mind.
Actually my current thinking over the last decade mostly aligns with what could be described as physicalist view of the reality, so even 'subjective' thoughts, ideas, concepts etc exist objectively in a physical sense as well (glia cells, neurons etc). (but that's a whole other topic ;)
I simply worded it 'real word' because I was attempting (perhaps ineloquently I will concede), to differentiate between frequentist and the Bayesian understanding of the term probability, because they differ [1].
Bayesian favors bringing in a priori beliefs into the model whereas a posteriori consideration of a problem, as occurs in frequentist approaches, favor isolation of the model.
>What will actually happen is, the universe splits into many "worlds"
Interesting, you state that so.. assertively :p I'd give the chance of a many worlds interpretation corresponding well with our physical reality, a low probability event, with a pretty high credibility interval ;)
[1] http://www.experiment-resources.com/bayesian-probability.htm...
Well, you probably guessed where I came from: http://lesswrong.com/lw/r5/the_quantum_physics_sequence/
I think most physicists agree that at the bottom, we have a distribution of "complex amplitude"[1] over a "configuration space"[2]. But as you can see from my second link, many (most?) physicists insist that we can derive a "probability" from a complex number. Note that such probability would then be an actual real world probability, where the universe itself is uncertain about what to do. True non-determinism.
It's only natural. At the experimental level, the researcher does observe Born statistics. Same setup, different results, so there is probability in the territory after all.
There's a problem with that however: The equations, which make such wonderfully accurate predictions, (i) are dederministic, and (ii) do not state at any point that the blob of amplitude we don't see disappear in a puff of smoke. They merely say that the blobs eventually stop interacting. The same way that if you launch a photon to outer space, never to meet it again, it won't disappear the instant it reaches the boundary of our observable universe. If you insist on a mono world, you have to assert that the other blob, despite being predicted by those otherwise accurate equations, somehow doesn't exist when you don't see it.
One way to do it is to believe that, contrary to what the equations say, the blob you don't see does disappear in a puff of smoke. Its amplitudes are literally zeroed out behind your back. In hindsight, this one looks nuts to me. I mean, how can we justify distrusting accurate equations in a way that doesn't even make experimental predictions?
Another way is to call the square moduli of those amplitude "probabilities", and pretend that because it's probabilities, the blob you are not in isn't real. But the equations do not make any difference between the two blobs. Then how come the other blob is less real than our own?
To me, those two explanations really feel bizarre. You have to start from a mono world assumption to come up with that. An easy mistake to make, since personal experience is telling us all the time that there is only one world. A bit like a leaf in a binary tree: its ancestors form a line, not a tree. But Kolmogorov complexity says a literal interpretation of the equations (which means many world) is simpler than anything else we currently know about. So to hell with personal experience (which by the way is responsible for much worse whackery than mono world).
Now there is a way out: we can admit that current physics imply many worlds, but insist that real physics probably don't. Current physics are not complete after all. We may have big surprises. This argument is certainly be much saner than the Copenhagen interpretation. So much that it does lower my probability for many worlds somewhat. Just not enough to squash my confidence. :-)
There is a theorem that if an experiment and observational apparatus are both quantum mechanical systems, then the many worlds hypothesis describes what happens when that experiment is observed with that apparatus. If quantum mechanics is merely a good approximation of some better theory, then to whatever extent it is a good approximation of the system, the many world hypothesis remains a good description of that interaction.
Therefore your confidence that the many-world's hypothesis is an inaccurate description of what happens when you observe the outcome of a quantum mechanical experiment is an insistence that your brain and body are not well-described by the best theory that physics has for how the world works.
What gives you that confidence?
You are criticizing me for trying to rely on data rather than a complex subjective model based on information from the beliefs of people that I have never met and have no input from? There is no way for me to attempt that approach that does not come down to some form of "making shit up".
More generally you are right that I prefer to work off of data rather than subjective opinion. Data I understand. Subjective opinion is valuable, but suffers from major potential biases. Correcting for that can be very hard.
Horizontal axis is how many engines failed, and vertical axis is how many times that number of engines failed in the simulation.
The code that generated this is:
from scipy.stats import *
from pylab import *
failures = [binom(9,p).rvs() for p in beta(1,1,1e5) if binom(9,p).rvs() == 1]
hist(failures, bins=9, range=(0,9))
show()
So the posterior probability on the entire rocket failing for uniform prior is around 25%. Uniform prior means that you believe that all single engine failure probabilities are equally likely: you think it's equally likely that engines fail with 10% probability as with 63% probability. Tweak the first two parameters to beta to change your prior belief. If you've seen n engines fail in your life and k engines succeed, then setting the first parameter to n+1 and the second to k+1 is a reasonable choice (so the current setting corresponds to not having seen any rocket launches prior to this one). For example if you've seen 2 engines fail and 99 engines succeed you use `beta(3,100,1e5)`: https://dl.dropbox.com/u/388822/rocketfailuredistr_for_optim... Hardly any probability mass left for entire rocket failure :)
An important detail is what happens when 3 or more engines fail simultaneously. Can they transform the mission in a soft landing or the "payload" is totally doomed? (For example, the Apollo XIII mission was a failure, but no one died.)
It isn't clear whether it was on the way to complete engine failure or it was shut down to play it safe.
At least I am explicit in the assumptions behind that model. They are:
1. A priori, all possible reliability numbers for a single engine are assumed equally likely. (This can be debated endlessly, but you need SOME prior for Bayesian analysis. If we had more data, then the prior would matter a lot less, but we don't so it does.)
2. Failure of engines is independent.
3. The rocket actually operates according to design parameters. That is it will survive the loss of any 2 engines, but not the loss of 3.
4. Past performance is a predictor of future performance.
Every one of those assumptions is questionable.
I invite you to create a toy model that takes that into account in some way and see what answers you come up with.
Short version: The right engine compressor failed due to apparent fuel ingestion, causing a major over-temp. The noise was mistakenly attributed to a blown tire, so the pilot left the gear down. This required MAX Afterburner on the remaining engine to recover, except the engine had a afterburner blowout, and didn't provide MAX power and the jet departed controlled flight due to low speed.
In summary, after the right engine failed due to fuel ingestion, the left engine had to push some air over to the non-functioning engine (for cooling I assume but it isn't stated). When the left engine afterburner did not light, it's "relight logic" did not trigger possibly because of the lower air amount. So the engineer who wrote the relight logic, assumed that the temperature would drop at a certain rate when the afterburner failed to light. Because the engine was working to assist the failed engine, that temperature drop did not happen and thus the afterburner did not attempt to automatically relight itself.
Sounds like it may be dependent after all.
This is your real error. SpaceX is effectively acting in perpetual “test” mode. New lessons are learned with each flight and each rocket test, and that greatly informs all follow-on operations.
The type of analysis you've done is reasonable for a fleet of deployed 747's. It is (forgive me) entirely useless for this sort of endeavour at it's current stage.
I wonder what your analysis of SpaceX's future performance would have been in Oct 2008, right after their first successful Falcon 1 launch and three failed priors?
https://gist.github.com/3855660
And the probability of any rocket engine failing, per launch, is 1 in 3!
In Bayesian analysis you have some set of prior theories with associated probabilities, you observe data, you alter your estimate of the likelihood of those theories, and then your next prediction becomes a weighted average of those predictions.
That number has no attached variance.
For instance one such interval has a MLE of an engine failure rate of 2.78%, a lower bound of 0.1% and an upper bound of around 15%. (Corresponding rocket failure rates range from 1 in 10 million to about 9.9%.)
How many failures are required to change your model from 2% to, say, 4%?
If 2 engines failed on the next launch, the model would predict a 4.8% chance of failure on the following launch.
As for your "build your model from a billion launches and 20 million failures" comment, if I had that much data, then it wouldn't much matter what reasonable set of priors that I started with, I'd wind up convinced that the true failure rate was very close to 2%.
Note that the prior that I am talking about is the distribution of possible theories before I saw ANY data.
First, you haven't really done a Bayesian analysis, for several reasons.
The easiest problem to fix is, you didn't specify any priors. We could validate your style of calculation by assuming single-engine failures are IID with parameter "p," uniform on the unit interval. (A beta distribution would be the standard conjugate prior.)
If we go ahead and make the uniform-p assumption, then what you've calculated is a most-likely posterior value of "p" (35/36). (This is the maximum-likelihood estimate for p.) But in a Bayesian world, "p" has a full posterior distribution, not just a most-likely value.
So, still in the Bayesian world, the probability of failure (P(N_fail > 2)) must be calculated on the basis of not the most likely p, but the posterior of "p". You didn't do this; you just used the most-likely p.
Which brings us to the second problem with your reply. You really can get a confidence value on P(N_fail > 2). You can get a full posterior distribution! It will be a 1-dimensional density on [0,1]. And you could calculate this, either by simulation or by an analytic procedure (because it's a low-dimensional problem).
This posterior on P(N_fail > 2) would be the answer to the parent's question. It would probably be rather "fat", validating the intuition that we don't have much data.
My reply might be a little smarty-pants. Sorry if it is! As I said, your analysis is a reasonable first cut.
Read the whole thing more carefully. I started with the calculation for the maximum likelihood estimate, but I ended with a prior with equal a priori likelihoods for failure rates 0%, 0.01%, 0.02%, ..., 99.99%, 100%. This is a reasonable discretization of uniform on the unit interval.
My reply might be a little smarty-pants. Sorry if it is!
I would suggest that before indulging a tendency to be a smarty pants, that it is good to read the whole thing.
You are still in error that there is not a way to describe the uncertainty in your estimate of the posterior probability of system failure. It has a posterior distribution, like everything else in a Bayesian analysis. You would compute it as I described -- Monte Carlo would be easiest.
If I have a normal coin and a coin which is double-sided (but I don't know which side), I'll give 50% for both of them coming up heads next time I toss. But if I toss them a hundred times each, my probability distributions for them look totally different.
I suspect this information is encoded in your priors, but I don't know offhand how to access it.
You can answer that question for each prior. The probability of, say, 3 failures is the sum over all priors of the probability that that prior is true, times the probability that it would leave you with 3 failures.
I leave writing a program to calculate this as an exercise to the reader. I've put enough time in on this one already, I have paying work to get back to.
Uniform prior, updated on the evidence that 1/36 engines have failed, I think this gives P(TFR = x) = x(1-x)^35 / (int x(1-x)^35 dx from 0 to 1). Apparently the integral is 1/1332, so P(TFR=x) = 1332·x·(1-x)^35. But that seems to have a mean of 5%, compared to your value of 2%, so I may have done something dumb?
In fact the failure of your neighbor likely affects the odds of your failure. So I'm being too nice to SpaceX.