An Introduction to Probabilistic Programming

thanks for this. there is also Probabilistic Models of Cognition [0] by one of the authors. I wish however that they stuck to Church language [1]

[0] https://probmods.org/

[1] http://web.stanford.edu/~ngoodman/papers/POPL2013-abstract.p...

does this require stats/probability knowledge?

on that topic, can anyone recommend an online stats/probability course? I tried the coursera one by Sebastian Thrun and couldn't get far into it because the "TA" examples were unintelligible.

The best example I've seen is the birthday paradox. How likely is it for 2 people in a N person class to have the same birthday? You can solve this deterministically using math pretty easily.

But if you modify the question to ask how likely it is for 3 people, or you add things like February 29th and leap years, or you add the fact that births are more likely during summer months, then it becomes extremely difficult to solve this deterministically. Instead, you run a Monte Carlo simulation to get approximate probabilities. This is much simpler to code and can be easily modified to fit new conditions.

A Monte Carlo simulation is probabilistic because you use random numbers in the simulation and you won't get a 100% perfect answer but you'll get close enough (and you can do some math to get error bounds).

You can generate a posterior distribution for any overdetermined problem. Basically, any scientific problem, as they must infer models inputs (parameteric or non-parametric) from observed model outputs (data).

Determinism is a nice illusion that quickly breaks down on real data. Scientific problems are non-unique and noise or inadequate models cause data and model to not mesh with one another. Deterministic answers have the property of being precisely wrong as opposed to mostly correct.

Rather than focus on the non deterministic random sampling, frame it as a programming language or library for manipulating and fitting statistical distributions.

An HTTP library is good for making web calls. A probabilistic program is good for estimating probabilities.

One example would be polling. You could write down a program that estimates votes from political polls. Then feed it polling data and get estimates of how people will vote.

It’s super interesting for combining scientific models with modern statistical computation in a flexible way

Also, randomized algorithms for all kinds of things are very useful in general-purpose computing

I wouldn’t classify floating point computation as deterministic

I wouldn't contrast it with machine learning or deep learning. Probabilistic programming is focused on building languages or libraries that incorporate fundamental probabilistic building blocks, model building statements and inference strategies as first class citizens within the language. Its been remarkably successful, check out Stan, Pyro, PyMC, Tensorflow Probability, JAGS, BUGS and Turing as very successful projects that have been used to tackle challenging and diverse problems with probabilistic modeling . The more modern probabilistic programming languages are actually designed to incorporate the advances of deep learning by making it easy to embed ANNs into models by using them to parameterize random variables, Pyro and Tensorflow Probability are probably the best examples of this.

note that these are not exclusive. you could divide ML into a traditional statistical approach and a probabilistic one that is concerned with deriving the underlying probability distribution. probabilistic programming is kind of like a domain specific language for achieving this. there is also differential programming that works on the same principle. there are certainly industrial usages of this paradigm. look up pyro (http://pyro.ai/examples/intro_part_i.html) for ppl and jax (https://github.com/google/jax) for differential programming

It's the standard tool for building Bayesian statistical models, which aren't glamorously crushing SotA records on AI-like prediction tasks, but are valuable in both academia and industry.

Not an either-or situation. Many ML algorithms can be formulated in a probabilistic way, and PPLs can represent them. But the choice of the underlying inferential algorithm (MCMC, variational, etc) greatly affects the scale and type of solvable problems, so it's not just the language itself.

PPLs using MCMC (e.g., Stan and pymc) are first-choice tools for Bayesian inference on small to medium data analysis, which is lots more common than Google-scale data analysis.

My professor in school years ago was working on probabilistic hardware and showing pretty good thermal benefits. It’s a drastically different way to build software so I don’t know how commercially viable it’ll become until someone shows it has a drastically better power per watt ratio (like 1000x or 10000x, maybe higher to account for the cost of transitioning existing software)

Probabilistic programming uses computer science techniques to do automated statistical modeling. For example, imagine I have a coin, and I want to discover if it is biased, i.e. if it lands on heads more often than tails. In a probabilistic programming framework, I can express my model as a simple Bernoulli model, `x ~ Bernoulli(p)`, and then automatically estimate the bias parameter `p` given some data (do "inference").

You can easily do this calculation by hand or in Python, but this does not generalize to more complex real-world scenarios. For complex probabilistic models, we must rely on numerical approximations. MCMC is just one algorithm for doing this approximate inference. Another popular technique is called variational inference [2]. Another commenter mentioned HMC [3], which is just a specific instance of MCMC.

[1] https://mc-stan.org/

[2] https://arxiv.org/abs/1601.00670

[3] https://arxiv.org/abs/1206.1901

Probabilistic programming can be done via MCMC approaches, but you can also infer the necessary quantities by using variational inference (which approximates the distribution described by your program with something that's simpler and easier to estimate).

Basically probabilistic programming is a way of describing a distribution, and then MCMC is one way of inferring the quantities in that distribution.

I'd argue that probabilistic programming is a language or framework of programming that lets you easily build and then fit probabilistic models to estimate or predict things of interest.

If you're taking a Bayesian approach to statistical modelling and inference, then they're probably a fairly good tool to consider. With the Bayesian approach you're trying to compute some posterior probability distribution that summarises your prior information (this might capture domain knowledge, information from related studies) and information from observations.

There are different ways to compute a posterior distribution. In very simple or contrived cases you might be able to manually grind out an answer analytically with pen and paper and lots of algebra and integrals. But that isn't very efficient or scalable. Also, nice algebraic structure is very easily broken by small perturbations to the problem statement -- need to add a weird bit onto the model to capture some real world behaviour? Good chance that ruins your algebraic structure and previous analytic "attack".

MCMC can be used to estimate the integrals you need when computing a posterior distribution. MCMC isn't the only way to estimate or approximate these calculations -- e.g. another approach is variational inference where a bunch of approximations are introduced to replace the original calculation with an approximation that is easier to compute -- this likely introduces bias into the results but can give you something that can then be solved analytically or semi analytically (e.g. approximate everything as Gaussian distributions and a lot of integration collapses to efficiently computable algebraic identities).

Some probabilistic programming platforms like Stan let you define your probabilistic model and parameters and decouple it from the computational backend used to estimate the posterior distribution. E.g. in Stan you can switch the computational backend between MCMC (https://mc-stan.org/docs/2_18/stan-users-guide/sampling-diff...) and ADVI (auto-differentiation variational inference).

MCMC has practical problems in that it is only guaranteed to give you the correct (unbiased) estimate asymptotically, in the limit if you run it for an infinite amount of time. If you're trying to approximate the integral of a function that is very multi-modal -- where it would be difficult for a global optimisation algorithm to locate the global optima -- then MCMC will likely also struggle to produce a good estimate. MCMC is difficult to parallelise effectively as the algorithm is inherently like an iterative local search procedure -- the next state in the chain is some mutation of the previous state. You can run n MCMC chains in parallel from n different initial configurations, but it's not obvious that you'll get a better estimate from n short chains vs a single long chain -- the longer a chain runs, the more chance it has of being able to discover and explore higher probability (more realistic, more plausible) configurations of the parameter space.

MCMC isn't only used for probabilistic programming, you can apply it for other things. E.g. it gets used in material science to study statistical properties of molecular dynamics simulations etc.

I think, and this is WAY outside my area, is that Hamiltonian Monte Carlo (a Markov chain monte carlo method) is used. Beyond that and I'm lost.

it can be used to build better statistical modelling and computational tools that might, in the future, enable scientists and researchers to more efficiently or effectively figure out statistical or causal relationships -- and in some niche applications, e.g. in pharma or finance or insurance or advertising, that information might be valuable and help some people get rich. but the people able to capture the most value from those developments are unlikely to be the researchers or the people building the software tools.