Comment by yummyfajitas | Hacker News Reader

Comment by yummyfajitas | Hacker News Reader

Parent Full thread

Simple Bayesian approaches take the opposite approach. You generally start with some relatively naive prior, and then treat the posterior as being the conclusion. Which is not very realistic if the real prior was something quite different.

I don't think this is a completely accurate portrayal of Bayesian stats. In Bayesian stats, there is no "real prior". Probability distributions are all subjective representations of belief. The prior is just what you believe prior to evidence, and the posterior is what you believe after you've taken evidence into account.

That said, moving away from p-values and towards something more robust is something the A/B testing industry needs. (Obviously I have my own opinion of what that something should be, and it's a bit different from what you are advocating.) There are far too many consultancies and agencies p-hacking their way to positive results ("hey unsophisticated client - guess what I made your conversion rate go up 25%!") and I'd love to see every one of them die.

There are different approaches a Bayesian might take. The one that I described is certainly among them, though it is not the only one.

I think the word "naive" is problematic here. Have you seen instances where Bayesians choose a prior that isn't at least somewhat informed by exploratory analysis?

A uniform prior would be one example. That's just saying "I've got no idea what the value is".

https://en.wikipedia.org/wiki/Prior_probability#Uninformativ...

There is a notion of an uninformative prior [0].

What would you do if you were studying a process that had not been previously studied, so you had no previous experience/data on which to base your a prior? If you wait, and choose a prior based on the data you observe then you are using an empirical Bayes method [1].

[0]: https://en.wikipedia.org/wiki/Prior_probability#Uninformativ...

[1]: https://en.wikipedia.org/wiki/Empirical_Bayes_method

It's quite common to choose a conjugate prior, which aids computation, and which is readily comprehensible within the discipline ("...and a Wishart prior for the covariance, of course..."). But which is, in effect, not at all informed by the data.

It's also common to have a complex, multi-level model setup which has a few hyperparameters (say, gamma distribution shapes) which are set basically arbitrarily. The idea being that the posteriors for the lowest levels (closest to data) will be learned/fitted, but the hyper parameters at the top-level of the model are fixed. This is very common in spatial statistics.

It's also common to make lots of (conditional) independence assumptions, just because they are convenient. Such as diagonal covariances, or conditional independence between "separate" elements of a model. But these independence assumptions are often just convenient or based on intuition, but not re-checked. Or, if they are checked and found wanting, it's "left for future work".

These practices are defensible, and attackable. But choosing convenient priors out of a bag of standard priors, without reference to the problem at hand, is very common.

EDIT: I forgot about the prevalence of uninformed priors. Thanks HN!

There is nothing more robust than p-values. The problem is that people call things "p-value of this experiment" when they aren't (for example, they may actually be talking about the "p-value of this other experiment" [which they possibly didn't even perform]).

Reply on news.ycombinator.com