Bayesian statistics and machine learning: How do they differ?

The distinction that the OP is looking for is generative vs. discriminative inference.

Generative inference: We know the process that generates our observed data and can model it to sufficient precision, s.t. we can establish a forward synthesis: Parameters in, synthesised data out. With that we can compare a synthesized dataset from a configuration of parameters with real observed data and, using Bayes (a.k.a. MAP estimation), infer the correct configuration of parameters that most likely generated our observed data, which is the answer we're looking for. Examples: Kalman Filters, Linear Regression.

Discriminative inference: We do not know the process that generates observed data or it's too difficult to model. In that case we ask machines to take a shot at modelling it. To do that we set up parametric data transformation pipelines (e.g. Neural Nets) and feed it lots of correct input/output pairs. Over time the model will learn how to transform the input (observed data) to the output (answer we're looking for) and hopefully generalise well when we feed it new input data that it hasn't seen before. Examples: NNs for classification.

Of course there are interesting mixtures and hybrids between the two and the lines are blurred in some cases but this is the general distinction.

For me, the difference between ML and most of what we use stats for in classical research settings (Bayesian or otherwise) has less to do with the specific methodology chosen, and more to do with the purpose of the exercise.

With ML, the utility of the model is largely a function of its predictive power. You want to accomplish some task. "Make it go"

With much of statistics in a research context (such as the social sciences as called out in the link), the interest is more on explanatory power of the independent variables. Most social scientists would happily trade a bit of predictive power for a more explanatory model that neatly maps back to a set of hypotheses. "Why does it go?"

For me, the gradual build up of (1) maximum likelihood estimation to (2) maximum a posteriori estimation to (3) full posterior approximation (or posterior sampling) was helpful to understand where Bayesian methods are in machine learning. Here’s a great video series by Erik Bekkers, who is at the University of Amsterdam. It assumes solid knowledge of calculus & linear algebra and takes you through the math and intuition of all fundamental ML methods: https://youtube.com/playlist?list=PL8FnQMH2k7jzhtVYbKmvrMyXD...

This is very close to my differentiation between the two.

Machine learning fundamentally cares about model performance on other data, like validation or test sets. It is looking for models that perform well, but not necessarily model the underlying process.

Bayesian statistics, like most statistics, wants to accurately estimate parameters in the model. It cares most about models that are portraying the underlying data-generating process.

This touches a little on philosophical difference, but I have found that one major difference is the quantification of uncertainty. Bayesian models, being based naturally around modeling a probability distribution, easily enables the researcher to make claims such as "I am 95% certain the mean lies between x and y, given the data." On the other hand, neural networks, decision trees, and other models more associated with ML than Bayesian statistics do not have this capability built in. (Though there are variations and techniques to build confidence intervals and such with them.)

I've seen the term "machine learning" used without criticism to refer to such a wide range of things that it seems pretty much synonymous now with multivariate discrimination and classification. At one time I would have said "machine learning" meant "deep learning models" but then it seemed to mean "computational multivariate discrimination and classification" and now the "computational" doesn't even apply.

I personally believe "machine learning" is a term that could be removed from use and nothing would suffer (and clarity might even be improved). I feel somewhat similarly about "data science" although at least that captures some intersection of database engineering and computational statistics that is useful when discussing very large data problems.

I choose not to focus on the differences between Bayesian statistics and machine learning. I choose to integrate the best parts of both.

If you are curious, the book "Probabilistic Machine Learning: An Introduction" by Kevin Murphy goes into greater detail.

In my non expert opinion it is difference in what model tries to predict, at least in some contexts.

For example in Reinforcement Learning P(R | A, S), eg. probability of reward given action and state, while in Bayesian network P(A | B, C), probability of even A given B, C distributions, in such networks you can do inferences.

I have no idea how correct or false this explanation is.

Reminds me a quote by one of my old professors:

"Before we started calling it nanoparticles, we used to just call it chemistry, but then we realised we got a lot more citations that way."

Machine learning is more broad than Bayesian statistics. More Bayesian statistics methods can be also be called machine learning, but less so the other way around.

Scikitlearn has a bunch of machine learning routines including knn, xdg, decision trees and so one. There is even a model zoo there. Id say bayesian mixed gaussian models to be in the same class of algorithms. Bayesian models have its own peculiarities and statistical foundation, but it isn't far fetched to imagine it inside scikitlearn and to be considered a machine learning routine.

The Probability -Data Science-ML space is so confusing. Any resources that would help clarify?