Thomas Bayes and the crisis in science
the-tls.co.uk
the-tls.co.uk
On the other hand, I think it somewhat exaggerates the extent to which Bayes' theorem and its associated work are rejected by statisticians, and also the extent to which a magical p > 0.05 limit was advocated. I believe it was non-statisticians who wanted easy tests that they could apply without understanding much about statistics, who were to blame in that case.
(Ok, I do computational Bayes for a living with natural scientists, so I'm inherently biased to view our work as shakier than it really is.)
> Variational Bayesian methods and deep learning for affective neuroscience, using probabilistic programming systems ProbTorch and Pyro.
It does sound like he's in for a rough time, epistemologically speaking.
0. E.T. Jaynes (2003), “Probability Theory: the Logic of Science,” http://bayes.wustl.edu/etj/prob/book.pdf [PDF]
Instead, an interpretation that seems more favorable to me, is that you consider all of the information at your disposal, that can be brought to bear on a problem, and this could include known constraints on likelihoods. Bayes's Theorem becomes a tool for working through problems where a single statistic can't be readily used to analyze an entire data set, e.g., when data come from disparate sources and can't be readily combined.
E.g., imagine you have an e-commerce site that has, historically, had a 2% conversion rate (landing page to purchase). Now you run an A/B test with two variants, a control (A) and a treatment (B). Both buckets get 10,000 landings, of which A converts 200 of them and B converts 250. How can you tell if A is better than B? Cutting to the chase, the frequentist approach (if applied naively) would be to model B as some distribution centered around 2.5%, for example N(2.5%, 0.15%) or Beta(251, 9751).
A Bayesian would say that this assumes a uniform prior - but that this is probably a bad prior because it ignores what we know about the historical conversion rate of 2.0%. Said another way, the above amounts to saying (before we run the test) that we think it's just as likely for B to have a conversion rate of 2.5% as it is to have a conversion rate of 100%. Clearly we don't actually believe this.
Why would that be wrong?
> A Bayesian would say that this assumes a uniform prior - but that this is probably a bad prior because it ignores what we know about the historical conversion rate of 2.0%.
What would a Bayesian conclude instead?
The issue is that modeling B with a distro centered around 2.5% ignores what we know about the historical conversion rate (2.0%) and the control bucket's conversion rate (also 2.0%). If our goal is to make the best estimate for the future that we can, we should take this data into account when evaluating B. As a thought experiment, imagine that you have A at 2.0% and B at 2.5% conversion for Week 1, with a historical conversion rate of 2.0%. Someone says they'll pay you $100 if you correctly guess what B's conversion rate will be next week, either (i) in the range 2.0% to 2.5%, or (ii) in the range 2.5% to 3.0%. I'd prefer to bet on (i) than on (ii).
> What would a Bayesian conclude instead?
One simple approach would just be to start with a more informative prior, like Beta(2+1,100-2+1) instead of Beta(1,1). This would pull bucket B's posterior distribution closer to 2.0%. Another approach is to use a hierarchical model [1], which will fit the individual buckets' priors for you.
[1] Here's something I wrote on this a couple years ago, more focused on solving multiple comparisons problems but with the same proposed solution: http://normal-extensions.com/2014/07/16/ab-testing-hierarchi...
Both the historical and the control bucket used version A of the website, and they are consistent in their 2.0% conversion rate. Version B is different, and it appears to have a different conversion rate of 2.5%. So why should it not have a future conversion rate close to 2.5%?
Let's replace the website with a 6-sided die. Historically, the probability of throwing a 3 was 1/6. Now you replace your die with a different die and throw it 10,000 times; the 3 comes up 2560 times. If I had to guess how many times the 3 comes up the next 10,000 throws, I certainly would bet that it's closer to 2560 times than to 1667 times.
> Someone says they'll pay you $100 if you correctly guess what B's conversion rate will be next week, either (i) in the range 2.0% to 2.5%, or (ii) in the range 2.5% to 3.0%.
Case A: The historical version A of the online shop had some influence on the conversion rate during the testing of version B, drawing the conversion rate of B down. This influence will fade away in the future, so B's conversion rate will be closer to [2.5%, 3.0%] than to [2.0%, 2.5%].
Case B: The historical version A of the online shop did not have any influence on the conversion rate during the testing of version B (compare the dice example above). Then both ranges are equally plausible. But "[2.0%, 2.5%] vs [2.5%, 3.0%]" is a bad dichotomy. A more relevant one would be "[1.75%, 2.25%] vs [2.25%, 2.75%]". In that case, I would bet on [2.25%, 2.75%].
> Both the historical and the control bucket used version A of the website, and they are consistent in their 2.0% conversion rate. Version B is different, and it appears to have a different conversion rate of 2.5%. So why should it not have a future conversion rate close to 2.5%?
It's all a matter of degree. You'd model B's rate as closer to 2.5%, but probably not centered around 2.5%. As you observe more data, the prior becomes less important. E.g., with 10k samples as in the original example, if you used Beta(2+1,100-2+1) as your prior, your posterior would be Beta(252+1, 10100-2+1) as your posterior, which is centered at 2.495%. But if you only had 1000 samples (and 25 conversions), you'd get a distro centered at 2.45%. And if you only had 200 samples (and 5 conversions), you'd get a distro centered at 2.33%. Etc.
> Let's replace the website with a 6-sided die. Historically, the probability of throwing a 3 was 1/6. Now you replace your die with a different die and throw it 10,000 times; the 3 comes up 2560 times. If I had to guess how many times the 3 comes up the next 10,000 throws, I certainly would bet that it's closer to 2560 times than to 1667 times.
In the case of a die where you believe any weighting of the faces is equally likely, this would be true. So this may be an appropriate model in this case. But in the case of the website, I don't think the conversion rates are equally likely, even for a new, un-tested site. If the historical conversion rate is 2.0%, and I'm forced to bet on the most likely conversion for a new (never before seen) variant B, I'd much rather bet on a number near 2.0% than a number like 99%.
> Case B: The historical version A of the online shop did not have any influence on the conversion rate during the testing of version B (compare the dice example above). Then both ranges are equally plausible.
This is exactly what I'm claiming is not true. It's not that A influences B, it's that A tells you something about the likely range of A and B (in this specific case of an e-commerce site). (The reason I chose the ranges [2.0%, 2.5%] vs [2.5%, 3.0%] is that if you model B independently, you'd be indifferent between these ranges; but if you use A to inform a prior, you'd prefer [2.0%, 2.5%].)
The smart approach is to have a set of a prior expectations, run through the data, and see what comes out the other side. Essentially, through a large amount of experimentations, you can narrow down what assumptions were indeed correct.
- Are security researchers grossly incompetent for failing to convince the IT industry to adopt best practices?
- Are voters in democracies grossly incompetent for failing to police their politicians and keep them honest?
- Are media outlets grossly incompetent for not squeezing out 'fake news', whatever your definition?
- Are epidemiologists and public health officials grossly incompetent for not convincing anti-vaxxers of their foolishness?
Anytime there is a breakdown in communications, both the listener and the speaker should probably shoulder some of it.
What would the proper use of statistics be? I believe it is to produce a Bayesian posterior probability distribution, but that wouldn't be the opinion of a frequentist.
Yes and no. I think most journal articles feature a copy-and-paste approach to statistics, reusing what other papers do, or worse, tinkering with test functions in excel. Yet, this seems to be ok with statisticians it seems.
From my personal experience applying statistics in the software world (and trying to build alerting models for monitoring applications, etc.) is, that since we have a lot of systems that continuously record date (instead of the way scientific studies usually work, collecting a set of data points, N=x), we get cross-validation of our results, which has drastically made me aware of the pitfalls of frequentist statistics.
I mentioned the Monty Hall problem to a friend (an engineer) and discussed the statical analysis done on this issue. (ie, statistically it's better to switch doors after the first is opened, as it improves your odds of winning slightly)
But he only answered with "nope, Bayes' Theorem says the odds are the same no matter what you do". The only other point he added was that if you do something only once, you have the same odds every single time.
I found this frustrating simplistic, because we _know_ through testing that this isn't true. That the odds are better if you switch doors.
Is this the fanaticism of Bayesians? He even called me a "frequentist" as if it were some kind of pejorative.
I've researched this issue, and even mentioned to him his logical flaw (you choose twice, not once) and still "Bayes theorum says there's no difference and you're wrong because you are a frequentist"
sigh.
the bayesian theorem is a mathematical fact. It's true if we just accept some basic axioms. Bayesian/Frequentist divide is also pretty independent from the problem, since it's purely math-based? We are not estimating something...In fact, an easy solution to the monty-hall problem is to just apply the base theorem and calculate the probabilities.
It seems simple, but the claim (not mine) was that the theorum states that since you are only choosing "once" you can't use an estimation, as that is based on collected data... which somehow doesn't exist or apply to you.
The reason is that somehow there are no odds/percentages/probabilities if there is only one instance of an event, or at least they don't change. (ie, 2 door, always 1 in 2 odds, period)
Normally I wouldn't detract from a thread like this, but participation seems low, so I can take the down votes in hopes of getting a response that helps me with this conundrum.
In the case of the Monty Hall problem, we have a complete understanding of what's going on, so there are no beliefs to update. There are no hidden rules or magic numbers we haven't been told about and we need to work out. You literally can't use Bayesian statistics if you understand a system completely.
Also, Bayesian statistics is a tool. It's not a law of nature. It can get the wrong answers. There is literally no guarantee that you will get the right answer when using it. Garbage In Garbage Out. It's just that there are cases where, empirically, it can be useful.
It seems then based on your reply that my friend is possibly wrong for invoking Bayesian theory regarding the Monty Hall problem for the exact reason he claimed it was the only answer? (note, my friend is a brilliant person and dedicated engineer who understands math far above my level. I only program with statistics and the PhD's tell me what to do.)
In other words, his argument is incorrect in his assessment because we actually _do_ have previous knowledge? (He pointed out that Bayes theorum was used for analysing insurance premiums for new kinds of insurance because there was no previous data to work with, which fits with your statement.)
This seems like too easy an answer for him to have missed, which makes me suspect of my understanding of your reply.
(yes, my initial question was serious, I appreciate you replying)
Don't confuse Bayesian inference with Bayes' theorem, which is a theorem that doesn't have any philosophy attached to it. Bayesian inference makes use of this theorem on top of some philosophical assumptions, which may be questionable depending on how you use them.
Bayesian inference also requires you to give your "prior" beliefs, which are the beliefs you have before you do the experiment. If these beliefs are strongly biased away from the right answer, you will get Garbage Out.
https://sc5.io/posts/how-to-solve-the-monty-hall-problem-usi...
I certainly don't expect you to research this or explain every detail for me, I may have to take a class on this subject after I retire to satisfy my need to grasp this. Thank you for time, this was enlightening.
It's very important to distinguish between bayesian inference and the bayesian theorem. The bayesian theorem can be proven and therefore must be true (if we accept the axioms). Bayesian inference has some real philosophical problems, since we often have no way to choose a justified prior. Frequentists approaches may also use the Bayesian theorem.
I always like the blog-posts of the angry statistician and he also has one about the monty-hall problem: http://angrystatistician.blogspot.com/2012/06/bayes-solution...
The important difference is that one is the probability of P(x == car), which is 1/3, and one is P(x == car | y == goat). Just apply the bayes theorem and calculate it yourself!
This is not true, unless by 'complete understanding' you imply that everything must be deterministic and we understand every single variable.
However, probabilities can be interpreted as statements of lack of knowledge at certain points of a process, even if the entire process itself is understood completely.
For example, if I have a process where randomly half the time I get a 55% heads biased coin, and half the time I get a fair coin, then I flip it 10 times, I can absolutely apply Bayes' Theorem here to figure out probability I had the fair or the biased coin after seeing the result of the flips.
This is a system that I understand completely, and part of that system is a single unknown bit, which we're trying to put bounds on.
A correct Bayesian treatment of the Monty Hall problem is given in a cousin comment: https://news.ycombinator.com/item?id=17438740
Thanks for giving me new search terms to consider. I have more arguments and examples to read through now. :P
Let A be the event that the car is behind door 1, B that the car is behind door 2, and C that the car is behind door 3. Let E be the event that the host opens door 3. We assume that the car is initially equally likely to be behind each door and that the host opens a door with a goat at random, never opening the door we picked.
Since A, B, and C are exhaustive and mutually exhaustive propositions, we can calculate the marginal probability of E by using the law of total probability:
P(E) = P(E ∧ A) + P(E ∧ B) + P(E ∧ C).
Bayesians like to define joint probability from conditional probability instead of the reverse; that is, define P(A ∧ B) as P(A | B) P(B) instead of P(A | B) as P(A ∧ B) / P(B).
So P(E ∧ A) = P(E | A) P(A). P(E | A) is 1/2 because we picked door 1, the car is behind door 1, and the host chooses at random a door that has a goat, of which there are two: 2 and 3. P(A) is 1/3. Therefore P(E ∧ A) is 1/2 × 1/3 = 1/6.
Similarly, P(E ∧ B) = P(E | B) P(B). P(E | B) is 1 because we picked door 1 so the host will not open door 1 and we assume the car is behind door 2 so the host will not open door 2, leaving only door 3 to be opened. P(B) is 1/3. Therefore P(E ∧ B) is 1 × 1/3 = 1/3.
P(E ∧ C) = P(E | C) P(C). P(E | C) is 0 because the host will never open the door the car is behind. P(C) is 1/3. Therefore P(E ∧ C) is 0 × 1/3 = 0.
So P(E) = 1/6 + 1/3 + 0 = 1/2. We know that the host opened door 3 (this is E), so the car cannot be behind door 3. How likely is it to be behind door 1? By Bayes’ theorem,
P(A | E) = P(E | A) P(A) / P(E).
We said earlier that P(E | A) is 1/2, P(A) is 1/3, and P(E) is 1/2. So P(A | E) = (1/2 × 1/3) / (1/2) = 1/3.
Given E, the car must be behind door 1 or door 2 since the host opened door 3. Therefore the sum of P(A | E) and P(B | E) must be 1. P(A | E) is 1/3, so P(B | E) is 2/3. The car is more likely to be behind door 2 than door 1. We initially picked door 1, so, if we want the car, we should switch.
Before Monty opens a door, so the prior probability, is 1/3 for each door.
After he opens a door, that door will have zero probability, as we know he opens a door without a car, then how to update the probabilities afterwards?
Seems the simple way to look at it is to not partition by door, but by chosen vs not chosen. Chosen is 1/3 and not chosen is 2/3 before and after Monty opens a door, so perhaps there is no “Bayesian information” revealed by opening the door anyway.
Right. That might be the easiest way for this problem.
More straightforwardly, without that shortcut:
- Call the doors a, b, and c. Assume, without loss of generality, that we choose door a initially. Let the random variable X ∈ {a, b, c} be the door with the car.
- Our prior probability is uniformly distributed: Pr(X = a) = Pr(X = b) = Pr(X = c) = 1/3.
- The data Y that we collect is our observation of which door gets opened by the host. The likelihood function Pr(Y = y | X = x) is the probability of the observation being y (i.e., that Y = y), given that the underlying state is x (i.e., that X = x). The only non-zero likelihoods are Pr(Y = b | X = a) = Pr(Y = c | X = a) = 1/2 and Pr(Y = c | X = b) = Pr(Y = b | X = c) = 1.
- Bayes' theorem, Pr(X = x | Y = y) = Pr(Y = y | X = x)·Pr(X = x)/Pr(Y = y), gives the answer, the posterior probability, which should be seen as a function of x. The denominator Pr(Y = y) = ∑ Pr(Y = y | X = x), sum over x ∈ {a, b, c}, is a normalising factor that makes the posterior probability distribution sum to 1. It is also the probability we, at the start of the game, assign to Y = y. In our problem, Pr(Y = a) = 0 and Pr(Y = b) = Pr(Y = c) = 1/2.
How about you put in the numbers and see if it comes out right or if I have made a mistake? :-)
Edit: Sorry, I just realised that we could have made it simpler by assuming that the host opens, say, door b.
First, get the person to realize that switching doors will always flip your initial outcome. If you picked a losing door, switching gives you the winning one. If you picked the winning door, switching gives you a losing one. There is no winning-to-winning or losing-to-losing. If you switch, you will always get the opposite of whatever you originally picked.
Once this idea has been understood, ask them to consider the probability that your original choice was a losing door. Once they agree that it's 2/3 (2 losing doors out of 3 possible choices), you can reply "that's right, and if there's a 2/3 chance that your current choice is a losing door, then there's a 2/3 chance that you're in a situation where switching will give you the winning door. Therefore, the chance that switching will give you the winning door is 2/3!"
Ah, so he was right about that, at least! :-)
Hence, the door he chooses to open is probabilistically informative about your initial choice. Treating the door he opened as information on which to condition does change the posterior.
I would say that they are based on two different view of what probability is, and that subscribing to one means that you cannot accept the other.
What's needed is a shift in researcher attitudes and incentives, to emphasize development of reliable knowledge instead of publication record. Just changing the rules of the game slightly will only lead to people adjusting their game slightly.
http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
Bayesian methods force you to actually think about how your data and model parameters are distributed, and explicitly specify a model.
Yep. I know some very senior scientists who get a lot of mileage out of finding places where "model-free" methods are implicitly assuming a bloody stupid model, and then attacking them.
While that's true, strictly speaking, it's akin to writing that, "We are living in a new logical age" when Boolean algebra was first finding wide application in engineering and the natural sciences. "Bayesian methods" just mean using statistical modeling techniques that conform to Bayes' rule as their normative guide. In "machine learning" or "frequentist" terms, it just means that good approximate-Bayesian reasoning minimizes the KL divergence between the true posterior and the approximate model (whether variational or by sampling or by training a neural network, whatever), as opposed to minimizing the classification hinge-loss or the mean squared error (though some of those losses have formally equivalent Bayesian priors).