My prior beliefs about the coin include the possibility that the coin has a bias other than 50%. This distribution may be centered at 50%, but I may assign more or less probability to extreme biases, and this determines how I update my belief when I observe several heads in a row.
Now, it is possible to do Bayesian analysis on probability distributions rather than "bare" priors, but that is just a way of assigning priors to a family of propositions rather than single propositions. For example, "This is a fair coin" is not a proposition to which one can assign a prior because it isn't a well-defined proposition. The intuition behind "fair coin" is something like "the ratio of heads to tails will approach 1 as the number of flips goes to infinity". But no coin can be flipped an infinite number of times, so this is a meaningless definition. Furthermore, consider a coin that always landed on the face opposite that of the previous flip. The ratio of heads to tails for such a bizarre coin would indeed approach 1 as the number of flips approached infinity, but that is not what is generally meant by a "fair coin". So the usual way of dealing with coins is assigning priors to the family of propositions, "If I flip this coin N times the probability that I will get M heads is X" for all possible values of N and M. But each of the propositions in that family gets a prior which is a number.
No, it doesn't. A continuous distribution is just a way of assigning probabilities to a parameterized set of propositions where the parameters are continuous and so the set is infinite. But for any given proposition in that set the prior is a number.
> Priors stands for prior distribution
No. Bayes's theorem is:
P(H|E) = P(E|H) * P(H) / P(E)
The "prior" is P(H) which is a probability, i.e. a number between 0 and 1, as are all the other "P" values. The "*" and "/" operations in Bayes's theorems are scalar multiplication and division.
A continuous distribution does not assign probabilities to each proposition but subsets of them. To see this concretely consider the continuous Uniform distribution from 1 to 1.5. It will for all values of its support have a probability density of 2. Most people would not consider 2 a probability.
For continuous distributions, Bayes's theorem becomes about probability densities and not probabilities.
Nope. It assigns probabilities to individual propositions.
> Most people would not consider 2 a probability.
That's true, but in your example 2 is not a probability but a probability density, and a probability density is not the same as a probability. To get a probability out of a probability density you have to integrate. For each possible interval over which you could integrate there is a corresponding proposition whose probability of being true is exactly the value of the integral.
(Note that there are all kinds of ways that statements can fail to be propositions. For example, you can't assign a Bayesian prior to the statement: "All even integers are green" despite the fact that you know what all the words mean.)
It is true that in your example, the outcome is binary: there is only a single proposition (the coin landed heads). And so the prior distribution is a Bernoulli distribution, but I would agree you could think of this prior as a single number between 0 and 1.
But in such a scenario, the only possibilities for updating beliefs based on evidence is to observe heads, then update the posterior to P(Heads)=100%, or observe tails, then update posterior to P(Heads)=0%. I might say that this is updating your beliefs based on evidence, but only trivially so.
If you observe heads and form a posterior of P(Heads)=99.999%, then it means that your prior probability distribution was more complex: it included the possibility that "the coin landed heads but I mistakenly observed tails". In this case, your priors include a family of propositions, not a single number.
In GPs example, the prior included the possibility that the coin was magnetic, etc. So it could not be represented by a single number.
No, it isn't. The prior on the outcome of multiple tosses is a Bernoulli distribution, but the prior on the outcome of a single toss is a number. And for a fair coin, that number is 0.5 before you see the result, and it is either 0 or 1 after.
Note that "This coin will land/landed heads" is a different proposition than "this coin is a fair coin." The latter is much more problematic because it is hard to define what a "fair coin" is.
> your prior probability distribution was more complex
No. You are confusing my prior on a single proposition (a number) with the set of priors on all propositions which I have considered in my lifetime. The latter is obviously a set of numbers, one for each proposition.
> "the coin landed heads but I mistakenly observed tails"
Yes, that is two separate propositions:
1. The coin landed heads.
2. My senses are reliable.
All evidence must be assessed relative to some "background model", a set of propositions whose priors are very nearly 1. And it is also possible to model an infinite family of propositions about continuous properties, e.g. The length of that rod is between X and Y for any two real numbers X and Y. But none of this changes the fact that for any single proposition the prior is a number.
> it could not be represented by a single number
That depends on what you mean by "it". The probability that the coin landed heads certainly can be represented by a single number, as can the probability that the coin is magnetic. Two different propositions, hence two different priors.