Terence Tao on the relationship between classical and Bayesian reasoning
google.com
google.com
Key points:
- in Bayesian reasoning, beliefs are not held as absolutes, but as probabilities.
- observations / measurements change those probabilities (using Bayes' formula). As a special case, certain measurements may set certain probabilities to zero; viewing only the "zero/nonzero" dichotomy is equivalent to classical logic.
- as a counterpoint to Holmes' "When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth", beware that you may have treated something as impossible which is merely highly improbable. There is a significant difference between actually zero and merely very close to zero.
- A implies B does not mean B implies A. But A makes B more likely does mean B makes A more likely; this is a consequence of Bayes' formula. (This does not imply causation. It's just a recognition of correlation -- if two factors are correlated, the presence of one increases the likelihood of the presence of the other.)
"Of course, this means to be really sure you have to use the slower algorithm or go through a rigorous proof. On the other hand, the probability that you'd mess up a proof is much higher than that."
It was a fantastic illustration of the principle.
Does Bayesian reasoning strike a difference between zero and asymptotically zero? It's my understanding that probability density functions are a method to dodge the problem in the frequentist approach, so I'm assuming the Bayesian approach also claims anything with probability limes 0 is impossible and just makes sure not to consider such cases.
"Zero" is a probability in the same way that "infinity" is an integer, because there's always a non-zero chance that your beliefs are mistaken.
The usual idea is to say that false=0 and true=1. Then one tries to model uncertainty with intermediate numerical values, replacing 2^n discrete possibilities with an n-dimensional space. This doesn't work and Bayesian reasoning does something else.
With n variables there are 2^n basic conjunctions. This will be familiar from writing out truth tables, or perhaps from not writing out truth tables because they contain too many lines. The Bayesian approach assigns a probability to every basic conjunction and thus lives in a 2^n dimensional space.
How can Bayes rule extend logic when it appears to be simpler? I think the answer is that it is working on a kind of "exploded" representation.
http://www.amazon.com/Probability-Theory-Logic-Science-Vol/d...
Available here: http://bayes.wustl.edu/etj/prob/book.pdf
The introduction/first chapter has a nice example about a policeman concluding a crime is being committed that's very relevant here.
"Chance" is a very frequentist word. It's Better to say "correlation makes us more willing to believe there is causation."