Comment by lisper | Hacker News Reader

Comment by lisper | Hacker News Reader

Parent Full thread

Of course it is possible to update beliefs based on evidence if priors are numbers. If I flip a coin but don't show you the outcome, you will rationally assign a prior of 0.5 to the proposition "The coin landed heads-up". If I then show you that the coin did indeed land heads-up (or tails-up) you will update your assessment of the probability of "The coin landed heads up" to something close to 1 (or 0).

Now, it is possible to do Bayesian analysis on probability distributions rather than "bare" priors, but that is just a way of assigning priors to a family of propositions rather than single propositions. For example, "This is a fair coin" is not a proposition to which one can assign a prior because it isn't a well-defined proposition. The intuition behind "fair coin" is something like "the ratio of heads to tails will approach 1 as the number of flips goes to infinity". But no coin can be flipped an infinite number of times, so this is a meaningless definition. Furthermore, consider a coin that always landed on the face opposite that of the previous flip. The ratio of heads to tails for such a bizarre coin would indeed approach 1 as the number of flips approached infinity, but that is not what is generally meant by a "fair coin". So the usual way of dealing with coins is assigning priors to the family of propositions, "If I flip this coin N times the probability that I will get M heads is X" for all possible values of N and M. But each of the propositions in that family gets a prior which is a number.

But priors are not *just* numbers between 0 and 1, as GP claimed.

It is true that in your example, the outcome is binary: there is only a single proposition (the coin landed heads). And so the prior distribution is a Bernoulli distribution, but I would agree you could think of this prior as a single number between 0 and 1.

But in such a scenario, the only possibilities for updating beliefs based on evidence is to observe heads, then update the posterior to P(Heads)=100%, or observe tails, then update posterior to P(Heads)=0%. I might say that this is updating your beliefs based on evidence, but only trivially so.

If you observe heads and form a posterior of P(Heads)=99.999%, then it means that your prior probability distribution was more complex: it included the possibility that "the coin landed heads but I mistakenly observed tails". In this case, your priors include a family of propositions, not a single number.

In GPs example, the prior included the possibility that the coin was magnetic, etc. So it could not be represented by a single number.

> the prior distribution is a Bernoulli distribution

No, it isn't. The prior on the outcome of multiple tosses is a Bernoulli distribution, but the prior on the outcome of a single toss is a number. And for a fair coin, that number is 0.5 before you see the result, and it is either 0 or 1 after.

Note that "This coin will land/landed heads" is a different proposition than "this coin is a fair coin." The latter is much more problematic because it is hard to define what a "fair coin" is.

> your prior probability distribution was more complex

No. You are confusing my prior on a single proposition (a number) with the set of priors on all propositions which I have considered in my lifetime. The latter is obviously a set of numbers, one for each proposition.

> "the coin landed heads but I mistakenly observed tails"

Yes, that is two separate propositions:

1. The coin landed heads.

2. My senses are reliable.

All evidence must be assessed relative to some "background model", a set of propositions whose priors are very nearly 1. And it is also possible to model an infinite family of propositions about continuous properties, e.g. The length of that rod is between X and Y for any two real numbers X and Y. But none of this changes the fact that for any single proposition the prior is a number.

> it could not be represented by a single number

That depends on what you mean by "it". The probability that the coin landed heads certainly can be represented by a single number, as can the probability that the coin is magnetic. Two different propositions, hence two different priors.

That framing actually stops holding for continuous distributions. Like in a distribution on heights of people there isn't a probability or prior associated with a particular height. Probabilities are assigned to subsets of this distribution (sometimes called sigma algebras). Priors stands for prior distribution and it really is best to see Bayesian analysis as a machine that consumes probability distributions as input and spits out probability distributions as output.

> That framing actually stops holding for continuous distributions.

No, it doesn't. A continuous distribution is just a way of assigning probabilities to a parameterized set of propositions where the parameters are continuous and so the set is infinite. But for any given proposition in that set the prior is a number.

> Priors stands for prior distribution

No. Bayes's theorem is:

P(H|E) = P(E|H) * P(H) / P(E)

The "prior" is P(H) which is a probability, i.e. a number between 0 and 1, as are all the other "P" values. The "*" and "/" operations in Bayes's theorems are scalar multiplication and division.

> No, it doesn't. A continuous distribution is just a way of assigning probabilities to a parameterized set of propositions where the parameters are continuous and so the set is infinite. But for any given proposition in that set the prior is a number.

A continuous distribution does not assign probabilities to each proposition but subsets of them. To see this concretely consider the continuous Uniform distribution from 1 to 1.5. It will for all values of its support have a probability density of 2. Most people would not consider 2 a probability.

For continuous distributions, Bayes's theorem becomes about probability densities and not probabilities.

> A continuous distribution does not assign probabilities to each proposition but subsets of them.

Nope. It assigns probabilities to individual propositions.

> Most people would not consider 2 a probability.

That's true, but in your example 2 is not a probability but a probability density, and a probability density is not the same as a probability. To get a probability out of a probability density you have to integrate. For each possible interval over which you could integrate there is a corresponding proposition whose probability of being true is exactly the value of the integral.

Ok I see where my confusion was. For you propositions are things that can be associated with measurable sets.

No, propositions are simply statements for which is it meaningful to assign a truth value. It might be possible to come up with a proposition that is associated with a non-measurable set, though I can't offhand think of an example. But remember: Bayesian probabilities are models of belief. Priors are peronsal. And so in order to assign a prior to a proposition, the statement of the proposition must have some referent in your personal ontology. You can't hold a belief about the truth value of a statement unless you know (or at least think you know) what that statement means. So unless a person has something in their ontology that corresponds to a non-measurable set (and I suspect most people don't) then that person cannot assign a Bayesian prior to a statement associated with a non-measurable set. But that's a limitation of that particular individual, not a limitation of Bayesian reasoning. For example: you cannot assign a Bayesian prior to the statement, "The frobnostication of any integer is even" because you don't know what a frobnostication is.

(Note that there are all kinds of ways that statements can fail to be propositions. For example, you can't assign a Bayesian prior to the statement: "All even integers are green" despite the fact that you know what all the words mean.)

Reply on news.ycombinator.com