Bayes Rules – An Introduction to Applied Bayesian Modeling (2021)
bayesrulesbook.com
bayesrulesbook.com
1) understanding what 'likelihood' actually is or represents
2) understanding what a 'partition function' actually is or represents
I always forget what these mean since I don't use bayes in anger
Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk
The probability of observing the data you observed, assuming that the model parameters take on a certain value.
> understanding what a 'partition function' actually is or represents
The probability of observing the data you observed, this time averaging over all possible parameter values (weighted by the priors).
Yes, in discrete cases. In continuous cases, you have to work with a probability density. I think this is one of the hurdles people encounter when they're first exposed to Bayesian stats. The probabilities, technically speaking, are zero.
The important insight in Bayesian work is that it's often not the probabilities themselves that matter but the ratios thereof, since from those alone you can compute posteriors.
In general, you have sums in the discrete case and integrals in the continuous case, but most formulas are otherwise the same.
The best explanation I can think of today is: Use the one that makes the math easier or more readable.
What is the age of a rock? That's not a discrete quantity: it could be anything.
I know the importance of continuous sets in the study of statistics as a branch of math. But I don't know of any measurements that can't be represented by integer multiples of a unit for all practical purposes. And the students in the non-calculus stats course can't grasp what continuity is anyway.
That's what the probability density specifies.
> But I don't know of any measurements that can't be represented by integer multiples of a unit for all practical purposes.
You can always discretize any real number, but why would you? I don't see how integers are easier to deal with than real numbers. Calculus can be viewed as the limit in which you discretize numbers infinitely finely. Once you know how things work in that limit, it's generally easier to use calculus than to work with discretized quantities. One example: summations are often more difficult than integrals, and one way of approximating sums is to turn them into integrals.
From my perspective, calculus is a basic part of mathematics that everyone should be expected to learn in school. In the US, calculus is often viewed as some sort of intimidating subject that only extremely clever people can grasp, and then only in late high-school or in university, but in East Asia, it's taught to children as a matter of course.
For example, given a coin with some probability f of getting heads and thus 1-f of getting tails (model parameter), a set of coin flips (data), and the assumption that each coin flip is independent of all others (model), the probability of seeing an arbitrary sequence with H total heads and T total tails after H+T flips is
p(H,T|f) = f^H(1-f)^T [0]
Note that this likelihood is normalized (i.e. sums up to 1) with respect to all possible sequences of flips of length H+T, e.g. for length 2, there are 4 possible sequences of coin flips:
p(hh|f) + p(ht|f) + p(th|f) + p(tt|f) = f^2 + 2f(1-f) + (1-f)^2 = 1
(Also note that the likelihood is only a probability for discrete data; it’s a probability density for continuous data, since the underlying terms would no longer be probabilities but rather densities. In that case, replace the previous sum with an integral ranging over the entire domain of your continuous data.)
Partition function (usually called a "marginal likelihood"): what if we instead normalize the likelihood with respect to the model parameters? Then it would no longer be a probability distribution with respect to the data, but rather a probability distribution with respect to the parameters. The marginal likelihood is just this normalizing constant. In the previous example,
p(f|H,T) = p(H,T|f)/<constant that would normalize p(H,T|f) to integrate to 1 over all possible values of f>
You can optionally weight this likelihood by a prior p(f), in which case the numerator would be p(H,T|f)p(f), with the partition function updated accordingly.
>Also the above reveal there is more elaborate structure in bayes than just the idea of 'updating your prior with new information', which is simply a platitude among STEM folk
I could not agree more. The core philosophical tenet of Bayesian inference is that this re-normalization of the likelihood returns something probabilistically meaningful. The debate over the validity of priors pales in comparison to the debate over whether the likelihood ought to be treated as proportional to a probability distribution over the model parameters.
[0] Note that this is distinct from the probability of seeing any sequence with H heads and T tails. That would require normalizing with respect to the number of total sequences with H heads and T tails (H+T choose H), yielding the binomial distribution.
Take the example used in chapter 2 of the book. It assumes that news articles can be classified as "real" or "fake", and there is no middle ground. It assumes that the initial prior produced by experts is reliable. And most of all it assumes that certain features, like exclamation points in the title, are causally related to realness or fakeness.
To illustrate this last point, consider analyzing the titles for a different features, the presence of the letter "z" rather than the presence of exclamation points. If it turned out that fake news in the corpus used to generate the priors just turned out to have more zees in their titles, would you then be justified in concluding that a news article about Zanzibar was more likely to be fake because it contained two zees?
That example might seem contrived, but if you use a Bayesian spam filter it is actually plausible that the presence of the letter "v" is dignostic of spam because of the prevalence of spam concerning viagra. But again, there is a causal model of this: viagra is a product that is often the subject of spam marketing, and the word "viagra" happens to have a v in it, which is a priori an uncommon letter in English.
But all this can fall apart depending on the circumstances. If you one day joined an email discussion of Stradavarius violins, the v signal could suddenly fail. An even more dramatic example: suppose you are an academic who starts to do research on spam filters and you have a collaborator who starts to send you examples of hard-to-filter spam. Now you have a very strong signal, but no straightforward textual analysis will allow you to extract it.
The hard part of Bayesian analysis is deciding what features to even look at. All the rest is borderline trivial by comparison.
I think the common argument is this is a strength of Bayesian analysis. Namely, that your priors make you explicitly state your assumptions. All models integrate assumptions, but not all of them make you explicitly quantify them like Bayesian analysis does.
I completely agree with you about hidden assumptions. But I don’t think that causal thinking is one of the problems here.
Bayes’ rule applies whether or not the relationship between X and Y is causal. Bayes’ rule is a predictive model and prediction does not need causation.
The problem in the Zanzibar example is that the model is misspecified. Or at least the model specification does not represent how you as a human being with cultural knowledge think about the problem. You look at Zanzibar and see an island in Tanzania. Your model looks at Zanzibar and sees two a’s and two z’s and a few other letters. So it is the wrong implicit assumption about the underlying model that’s giving you “non-sensical” results.
Yes, that is exactly my point. The way "you as a human with cultural knowledge think about the problem" is a major component of Bayesian analysis. The math is almost incidental. By the time you have chosen what features to do the math on, most of the heavy lifting has already been done.
This matters because if you just crunch the Bayesian numbers on an arbitrary data set you will almost certainly find features that appear predictive but are not. The more numbers you crunch, the more likely this is to happen. A lot of people lose money in the stock market this way.
This sounds extremely plausible to me. I'd like to use it as an example in something, any data to back it up?
Try to imagine that there's quite a few experts around HN ...
This is what I was responding to:
> This matters because if you just crunch the Bayesian numbers on an arbitrary data set you will almost certainly find features that appear predictive but are not. The more numbers you crunch, the more likely this is to happen. A lot of people lose money in the stock market this way.
Please explain how this is not exactly the same concept, expect with different motivations for looking through the data.
> Try to imagine that there's quite a few experts around HN
Why be a jerk though? I gave a reasonably well thought out comment related to one I found interesting, and made sure not to overly imply expertise.
"P-hacking" applicable to vastly more than stock data and is not evidence of anything, evidence is what was asked for.
Your response implies I'm ignorant and can't google p-hacking without being told the name of that one way of finding false correlation applicable to many, many fields. Now you're getting a bit cross about it that I've quietly pointed out that it isn't useful, and you're not actually providing expertise here which you thought you were. It's kind to provide expertise and I appreciate the kindness. You misread, fine. Best.
edit: Oh wait
> Or maybe that term is more general, I'm not 100% sure, but it's close.
That is an edit addition to the parent after my reply, so I guess you did get the point. Best.
[1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
I’m resistant to it because some analysis I do (say, inferring atmospheric composition from the spectra of light you shine through the atmosphere) does fit rather well in the Bayesian setup. In a very real sense, if you are doing a bunch of these inversions and not using Bayesian methods, you are leaving RMSE on the table.
Here, the “features” are the spectra you sense and you add some basic prior information based on historical expectations. You don’t have that much wiggle room to invent new variables to confuse the analysis.
But the situation changes if you have woolier problems with human judgement in finding “features” — the variables in the prior or likelihood in the spam example you give. And you can fool yourself badly!
E.g., to go with your market example, I have two models for the stock market, and 1 month of historical data strongly favors model 1 over model 2, so I bet heavily on model 1 as if that’s the only possibility. But it wasn’t, and I lose.
The area of causal inference uses a slightly different set of tools that gets outside the Bayesian setup, including the Do() operator, which allows for manual intervention to probe the linkages in systems like these. There’s a lot of activity there as we’re looking for systems that learn reliably.
I don’t know if you listen to podcasts, but Judea Pearl was recently on Sean Carroll’s (excellent) Mindscape podcast. They discuss this at some length, it was very nice.
https://www.preposterousuniverse.com/podcast/2022/05/09/196-...
PS: it’s enjoyable to see your comments here.
I don't know much about atmospheric science, but the canonical "real" example of Bayesian reasoning is a diagnostic test for some disease, where you are told that X% of the population has the disease, where X is typically a small number, so the "surprising" result is that your odds of having the disease are low even if you test positive and the test is generally accurate. The problem is: how can you possibly know what percentage of the population has the disease? To determine that you'd need a reliable test, but there is no such test. That's the whole point! If there were a 100% reliable test you wouldn't need to do any math.
For your disease example -- if you don't know the accuracy of the test ("the model"), and/or if you can't figure out what percent of the population has the disease ("the prior"), then you can't turn the Bayesian crank.
Yet, some do, because Bayesian good.
... dates back to antiquity. I'm not a statistician, but my impression is that Bayesian methods are a way of formalizing that approach. On the other hand, use of the term "prior" is somewhat misleading because the formulas do not specify a time sequence for acquiring information.
I tend to have a lesser view of sprinkling Bayesian terminology into blogs about social issues. That's what I call Bayes Theater.
I honestly don't even mind paying for them but I want the flat file.
wget -r https://www.bayesrulesbook.com/ && cd www.bayesrulesbook.com && pandoc --pdf-engine=xelatex foreword.html preface.html chapter-{1..19}.html references.html -o book.pdf
For advanced reader this is really shallow, for beginners its really advanced (who really want to read 543 page book as beginner).
> Bayes Rules! brings the power of Bayes to advanced undergraduate students and comparably trained practitioners. Accordingly, the book is neither written at the graduate level nor is it meant to be a first introduction to the field of statistics.