Comment by tjradcliffe | Hacker News Reader

Comment by tjradcliffe | Hacker News Reader

Parent Full thread

One aspect of the Bayesian/Frequentist "controversy" (which amongst working scientists isn't very controversial) is the supposedly subjective nature of probability under the Bayesian system.

I think this is mostly a conceptual error on the part of Bayesians: we should be treating the "degree of belief" in the Bayesian system as precisely that, a degree of belief, and nothing more. It is not a "probability", which I'm happy to leave with a frequentist definition.

That is, Bayes' theorem should read: p(h|e) = P(e|h)*p(h)/P(e)

where the lower-case "p"'s indicate plausibility (degree of belief) and the upper case "P"'s indicate probability as measured by frequency of occurrence.

Probabilities under this system are completely objective. Plausibilities are--naturally--subjective. As someone who advocates for an epistemology of a knowing subject (that is: there is no knowing without a subject that knows, and the central question of epistemology is how that subject knows) it is entirely natural that the degree of belief in a proposition should be subject-specific. After all, whose degree of belief is it?

This is not a failure of the Bayesian approach, but a virtue: it allows us to successfully navigate the many traditional paradoxes of subject-free theories of knowledge, which put "truth" outside of the realm of the knower and pursue a false goal (certainty) much like pre-scientific alchemists pursued the false goal of the philosopher's stone.

One issue with this: calling the likelihood P(e|h) 'objective' ignores the fact that the choice of model is usually quite subjective, and often affects the posterior to a much larger extent than the choice of prior, especially if you're working with a lot of data.

> the choice of model is usually quite subjective

Actually it's not. That is one of the most remarkable things about the universe we live in: it's really hard to come up with compact models that are consistent with all the data. So you can eliminate the vast majority of prospective models by eliminating those that 1) aren't consistent with the data or 2) don't compress the data efficiently. What you are left with is invariably a very small number of potential models.

The bigger problem is people's unwillingness to let go of their cognitive biases. For example, economists still (AFAIK -- I don't actually read the economics literature) make their careers publishing papers based on the efficient market hypothesis despite the fact that there is overwhelming evidence that markets are not efficient. There's no getting around the GIGO principle.

There are domains, like particle physics, where models are quite strongly constrained by data. And then there are domains, like climate science, economics, political forecasting, genetics, sociology, etc., where the underlying phenomena are very complex, data are noisy, and quite a wide range of models can be made to fit the available data. The latter sort of situation is becoming more and more common, as machine learning and statistical modeling are increasingly being applied to human behavior and other such messy areas that fall well outside the realm of 'clean' physical laws. So I submit that subjective model choice is, in practice, a very important part of a statistician / scientist's art. (as is understanding the nonstatistical reasons why one might choose a 'worse' model over a better model, e.g., computational tractability).

"Subjective" means "based on or influenced by personal feelings, tastes, or opinions." Whether or not the choice of theories is subjective on this definition depends on whether or not you care about the truth. Consider:

Proposition P: Anthropogenic climate change is a myth.

Suppose I subjectively choose a Bayesian prior for proposition P to be very close to 1. I can turn the crank on that and come up with models that are consistent with all the available data. Of course, this will lead me to conclusions that some would consider bizarre ("conspiracy theory" is the pejorative term) but because my feelings, tastes and opinions are just as valid as anyone else's (qua opinions) there is no subjective basis for preferring or rejecting my conclusions. But, of course, there is an objective basis for rejecting my theory because it has more explanatory loose ends than a theory that doesn't have to invoke conspiracies.

Perhaps you want to use subjectivity to choose among theories which are objectively "equivalent" in some sense, i.e. similar explanatory power, agreement with the data, etc. There's nothing preventing you from doing that, but what would be the point? The best you can say objectively is that there is no basis for preferring one theory over another and there's more work left to be done (c.f. string theory). You could pick one theory over another because you think it's "prettier" but so what? If there was anything actually to be gained then (again, by definition) you could have made the selection objectively, not subjectively.

Wait: the entire point of Bayes's idea was that under very sensible and straightforward rules, "plausibilities" behave exactly like probabilities. So why multiply concepts?

What is the difference, in your proposal, between "plausibilities" and "probabilities"?

You are correct: the way these mathematical creatures work, plausibilities behave like probabilties. What tjradcliife is trying to do, by differentiating between plausibilities and probabilities is to address the concern of frequentism with Bayesianism: namely, the "P(h)"-- what is usually termed as a "prior probability" in statistical jargon-- usually does not arise from objective data.

Accordingly, Bayesianism can be treated with suspicion-- since the P(h) is arbitrarily specified, one could "load the dice" and obtain a P(h|e) ("posterior probability") guaranteed to suit the analyst's conclusions.

By differentiating the two creatures, tjradcliffe hopes to resolve the dis-ease with calling P(h|e) a "probability". By frequentist philosophy, probabilities should be grounded in some sort of firm soil, e.g., the relative frequency of what would be observed if one could conduct an experiment ad infinitum. A Bayesian "probability" is not grounded this way, so calling it a "plausibility" instead would resolve this tension.

Another way of putting this-- P(h) to a frequentist feels like a tainted probability-- really not a probability at all-- so even after updating the prior probability with data [i.e., P(e|h)/P(e)], the posterior probability P(h|e) still retains the influence of that original taint. Solution: stop calling the Bayesian probabilities "probabilities"; instead, call them "plausibilities".

Well, one problem is that some people, including myself, object to assigning probabilities to facts of nature. What is the probability that the mass of the Higgs Boson is 126 GeV +- 0.1 GeV? That doesn't make sense, it is either 100% or 0%, true or false, as the mass is a natural constant. It's like asking for the probability that 3 is between 2 and 5.

Of course, you can make statements like P(m=126) sensible, by recognizing P as degree of belief, or plausibility. These don't describe facts, but rather knowledge, or your uncertainty. The wonderful thing is that they fulfill the same axioms as ordinary probabilities, and thus you can use Bayes theorem and so on with them.

Distinct from degrees of belief are probabilities (or frequencies in some Bayesian literature). These do not describe your uncertainty, but are really physical properties, or facts of reality, in disguise. If you have a fair die, P(1) = P(2) = ... = P(6) = 1/6, the equality of the numbers means that all faces have the same area (under the plausible assumption that the die has a uniform density). The area of a face is (for small deviations from P=1/6) proportional to the probability to land on that face. So a probability of P != 1/6 means you have a deviation from the perfect cube form.

One reason I think it is very important to be aware of this distinction is that people object to "Bayesianism" because most Bayesians don't make the distinction. People do not reject Bayesianism because it's methods don't work, they reject it because it seems philosophically unsound. If they would stop calling plausibilities probabilities, a lot of discussion and confusion would go away IMHO.

(Maybe we should forgo all the p words and call the Bayesian quantity just "awesome score" or something. Less people would have problems with evaluating an arbitrary formula that doesn't have all the philosophical and political connotations that probabilities have.)

one problem is that some people, including myself, object to assigning probabilities to facts of nature

What do you feel you gain from this philosophical objection? Is there a difference between a "fact of nature" and a one time event that has already happened? Consider a coin flip that has happens a light second away. After the flip, but before the result has reached you, much like a physical constant, isn't it also 100% heads or tails but just unknown to you? Does it change from being a probability to a plausibility at the time of the flip?

(asked with ignorance but genuine curiosity)

That's actually a good question. I don't know if you gain much from that objection besides clarity. After all, both "kinds" of probabilities share the same mathematics.

I think it's the other way around. People have certain intuitions, or prejudices if you wish, and it's useful to have terms that match your intuitions - even if they are slightly redundant. I find it helps me understand and explain statistical problems better.

About the coin: I think you have to distinguish between one concrete flip of the coin, and the coin itself. The coin itself has certain probabilities for generating heads and tails, and they don't change.

A concrete flipping of the coin is a different situation. You might see the coin, and know with near 100% certainty what it shows. Or you might have some machine which detects whether it is heads or tails, and this machine has a certain accuracy. When it signals heads, you might say it has a 95% plausibility to really be heads. And before you measured the state of the coin, you might say the plausibility is 50%, assuming the coin is fair. So the plausibility does change at time of observation (or when you get new information).

What about the time of the flip? Well, after the flip has occurred, this concrete instance is either 100% heads or 100% tails of course. If I repeat this exact coin toss (which I can't do in reality), I will always get the same result. And the probability to have heads, when you have heads, is 100% (P(toss #1 is heads | toss #1 is heads) = 100%).

This statement is a bit silly, but I think it shows that the probability of a thrown, lying coin to be heads or tails is not a very interesting quantity. Our degree of belief, given certain facts, that it is heads or tails, is interesting. And the probability of the coin in general, determined by its geometry, is interesting. And especially interesting is how you get from one to the other, which is where Bayesian inference, hypothesis testing, and so on come in (The coin is known to be rigged with heads=99%. My flawed detector says it is tails. What should I believe is the state of the coin, or what should I bet on?).

Finally, I should note that you should take what I've said with a grain of salt. I'm at best an armchair statistician, although I do get to think about stuff like this a lot at work (as a physicist).

I have to thank you a lot for posting this. This is exactly what I've been saying for a while, especially the distinction between probabilities and plausibilities (degrees of belief). I wasn't sure if this is just an idiosyncrasy of mine, or if someone else uses this distinction (see a question of mine on StackExchange [1]).

Anyway, thinking this way takes the wind out of the sails of many arguments in the Bayesian/Frequentist debate. Degrees of belief are not identical with probabilities ("frequencies"), but they are very useful, and you should have both in your toolbelt. Trying to use only one and calling yourself a Bayesian or Frequentist is very limiting in the end.

[1] http://physics.stackexchange.com/questions/32757/probability...

Reply on news.ycombinator.com