Comment by alexpetralia | Hacker News Reader

Comment by alexpetralia | Hacker News Reader

It appears to me that the reason Bayesian probability is somewhat elusive is due to its metaphysical underpinning: what do we mean by probability?

If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty.

Then, what is probability? If everything will happen with 100% certainty, but probability certainly appears to exist, then probability must reflect something about our information about something occurring.

The author refers to two foundational approaches to our state of knowledge. The first is the objectivist approach, which states that everyone who has the same state of knowledge about a system will evaluate the same probability of something occurring. The second is the subjectivist approach, which states that a given individual with a certain state of knowledge will evaluate some probability of something occurring. To me, these appear to be the same thing except insofar as the former requires a consensus of many while the latter a consensus of one.

The author asks how we might actually define Bayesian probability without resorting to the frequentist approach (i.e. hypothetically simulating many trials of the same event, however infrequent in reality it may be).

First, he says this would mean "interpreting [the credible interval] like a confidence interval". I am no statistician, but is that necessarily true? I don't see why confidence intervals would suddenly emerge in order to interpret a credible interval.

Second, I am not sure the frequentist interpretation is so problematic. When we interpret the plain-English definition of a probability, are we not mentally simulating repeated trials in order to evaluate something's occurrence? What else could a probability imply? If something has a 20% chance of occurring, then it does not occur 80% of the time, and so we must envision 80% of universes (part of the hypothetical trials) where it does not occur. I don't see any other way around this, frequentist or not.

(Note: I am not a statistician, while the author is, and the above is simply my layman's understanding of the article.)

There's no definition of probability that doesn't involve philosophy or metaphysics [0]. Calling frequentist stats "objective" really bugs me. There is no such thing. Every inference procedure involves subjective choices.

Of course, frequentist and Bayesian stats are completely mathematically equivalent. The choice just affects our mental patterns.

[0] https://en.wikipedia.org/wiki/Probability_interpretations#Ph...

No, frequentist and Bayesian statistics are not equivalent.

There are some special cases in which a frequentist 95% confidence interval and a Bayesian 95% credible interval based on some sort of default prior are numerically the same, but that doesn't happen in general.

Statisticians would hardly have been vigorously debating the issue for two centuries if it didn't really matter.

From the article:

>>>> Let’s now suppose that we’ve done a Bayesian analysis. We’ve specified a prior distribution for the parameter, based on prior evidence, our subjective beliefs about the value of the parameter, or perhaps we used a default ‘non-informative’ prior built into our software package.

At first blush the difference is that the Bayesian is using more information. Now don't get me wrong, if Bayes theorem and its progeny give us useful tools for incorporating that information in our analyses, so much the better.

It's been only one century since Ronald Fisher misunderstood statistics and thereby created what's now known as 'frequentist statistics'.

Every frequentist technique has a Bayesian interpretation and vice-versa. Confidence intervals are equivalent to credible intervals with certain priors.

The result of Bayesian inference is a whole posterior distribution, not just a confidence interval. Any attempt to produce a frequentist version of a posterior distribution is either going to end up just being Bayesian inference in disguise, or be inconsistent.

But even if we focus just on confidence intervals and credible intervals, there needn't be the equivalence you state. A comment elsewhere here discusses a ridiculous confidence interval that is either the whole parameter space or the empty set. That's never going to be what a Bayesian credible interval gives you.

I don't think you're engaging in the specific sense of objective (, subjective) meant here.

A frequentist interpretation of probability is objective in the sense that it grounds the probability value in objective features of the world.

A bayesian is subjective in that the probability valuation of an event is grounded in the belief-state of its observer.

My point is you that you cannot ever actually claim to ground a probability value in objective features of the world. Many events aren't repeated, for example.

Sure you can, frequentists do not ground probability in repetition. Repetition is just a measurement process.

Their grounding is some real physical parameter, eg., with the coin the geometry of it having two sides.

I think a further problem may be that probability could take on different meanings under different applications. I admit that I may be influenced by how I learned statistics -- in a math class that was primarily focused on proofs and not applications. But I've formed the view that math is math, we choose a math technique that works for the situation at hand, then we choose an interpretation that works for guiding and explaining what we're trying to do.

It's just a perspective, but Bayesian inference makes the prior explicit, wheareas it is implicit with frequentist inference. You can't not have a prior.

That's fair, it forces you to document your assumptions. But at some point as you drill down into assumptions, the trail will grow cold. The article even talks about inserting priors that are either subjective or some default function provided by the software.

A possible third approach is to create a hypothetical model and feed random data through it, to get an idea of the spread of the outcomes. Modeling doesn't stumble on conditional terms, and if you're unclear on an assumption, it won't run. The computer doesn't know whether it's a frequentist or a Bayesian, or something different from both.

I'm not a statistician. Whenever I need to do something with statistics, I always test my computation with random data.

In fact I wonder, if modeling had been possible since the birth of statistics, if we would even bother with things like elaborate formulas for statistical tests.

Bayesian stats actually let you integrate stochasticity into deterministic models fairly easily, in a way you can't really do with frequentist stats. Bayesian methods are common for geophysical inverse modelling, for example. Probabilistic programming does this, for example. Exact Bayesian inference is impossible in the general case, but approximate inference often works well.

Subjective priors are one of the main advantages of Bayesian stats. Regularization used in ML corresponds to using subjective priors, for example. L2 regularization finds the MAP with a normal prior, favoring parsimonious solutions.

It seems a Bayesian interpretation of probability is more general. The frequency of events over an infinite number of trials is one way of interpreting probability for things that are able to be repeated. But this wouldn’t make sense to apply for an election that is only going to happen once and yet one still wants to be able to quantify uncertainty in these situations.

There are a few things intermingled in this election example.

1. The outcome of the election here is not a probability. It is the population value - the ratio of people voting for candidate X on the election date. It doesn't have to be repeated in the same way measurements of height for all people in United States would not have to be repeated, if instead of vote we were measuring heights.

2. Frequentist probability doesn't require to physically repeat things. It can reason about what would happen in the repeated sampling under certain conditions, and then draw inferences about those assumed conditions. With the election example: if you get a survey of 100 people with 70% voting for candidate "A" we don't need to repeat this survey in order to know the likelihood (frequency) of this result happening if the real proportion of people voting for candidate "A" across the US is 50%.

If you’re trying to quantify your uncertainty about who will win the election, a poll would only be part of it. You want to be able to combine disparate sources of information. Maybe there is preference falsification and you want to incorporate as some sort of prior. As things get further from simple sampling from a population the frequency interpretation makes less and less sense to me.

Sorry, I don't see how this reply is related to the original objection that frequentist methods are not applicable to events that happen once.

You’re correct that probability still works in a hypothesized deterministic universe. It’s a point that’s often too often forgotten, causing discussions to go down unnecessary rabbit holes debating foundations of quantum mechanics when discussing the roll of a six sided die.

Statisticians and mathematicians have gone very far down the path you’re discussing, and you might be interested in some sets of axioms that have come up around probability and relaxations of true/false logic.

The Kolmogorov axioms [0] are the “standard” probability axioms, and are phrased in terms of set theory and measure theory (not requiring any mention of physics or a physical universe!).

There are other ways to quantify degree of belief, however, and they are very interesting. Apparently Cox’s theorem [1] justifies a popular probability framework for Bayesians. But there are many more interesting ways to do degree of belief, like Dempster-Schafer theory [2], which I understand to be a plausibility calculus.

Everybody seems to find a single system and decide it’s the only one out there,

[0] https://en.wikipedia.org/wiki/Probability_axioms

[1] https://en.wikipedia.org/wiki/Cox's_theorem

[2] https://en.wikipedia.org/wiki/Dempster%E2%80%93Shafer_theory

> If we live in a materialist deterministic world - which many would cite as an axiom for simplicity - then there really is no probability. Everything happens with 100% certainty.

I don't know who "the many" are - but I thought determinism had already been disproved.

I am not a physicist so I will not go into quantum mechanics - but I will take a simple example from Science Fiction, and that is the Temporal Paradox. https://en.wikipedia.org/wiki/Temporal_paradox

How does this disprove determinism?

Sorry - I was being flippant because your argument involved time travel. After the fact an event has 100% probability that it did occur but before the event it does not.

The entire field of chaos theory is to make chaos deterministic so you are in good company. There is no general mechanism (yet) to do this. Quantum mechanics is the most interesting area.

The measure of determinism is that it predicts the future before it happens. People have been trying to do this for years in weather and the stock market. This is where the concept of Chaos came from.

Leaving math and physics - there is philosophy. To apply determinism to people you would have to decide there is no free will. Maybe this is true and maybe not.

Reply on news.ycombinator.com