You got this new kind of variable. It’s special because instead of holding an object or a number it holds the dice. When you evaluate this variable it gives you a dice roll.

To use it to solve real systems you create a model where some of the variables are dice and you shake the system until you get an answer that satisfies you.

Consider the following (traditional) program:

  x = raw_input()
  x *= 2
  assert x >= 10

Now we can ask the question: "Assuming the assertion passed, what can we say about x?" In this simple example, we know that x >= 5, but in general the possible values of x may be much more complicated.

This is the kind of question probablistic programming is designed to answer, except instead of being an arbitrary, unknown value, x is specified as some distribution (say Gaussian). Then, the question becomes, "What is the posterior distribution of x, assuming all of the assertions pass?" In other words, figure out how likely different choices of x were, taking into account both the prior (i.e., that x is Gaussian) and the new information from the assertions.

For a simple example of why this is useful, suppose we have a program that generates random images (this is our prior). We also have some real photographs. We can assert that the random image generator should have a pretty good change of generating the real photographs. Then, the probablistic program "execution" will try and compute a new generator that creates more realistic images.

I honestly don't think it would be possible for a 5 year old to learn or even grasp the concept of probabilistic programming. So maybe wait until high school or if very gifted middle school where you have a more solid mathematical foundation. It's the process of inferring the parameters of a Bayesian probabilistic model from data and then making predictions with that model.

A probabilistic programming language lets you express the model in code, and then it performs inference automatically using various methods. Note doing it manually may produce better results but it is a very involved and time consuming process.

My impressions after familiarizing myself with Stan, BUGS, JAGS, and a few Python libraries for Bayesian inference:

Probabilistic programming languages are designed to specify and evaluate statistical models. They usually sit somewhere between declarative and imperative languages. They're basically DSLs in the sense that they come with built-in primitives for various statistical distributions and linkage functions, and the model is sometimes but not always lazy evaluated, but they also often have procedural components like variable assignment and loops, and in some software like Stan, the model has to be specified in a certain order since it is evaluated procedurally.

The languages are also de facto coupled with the engines that run them -- I don't know of any probabilistic languages that have been formalized without a corresponding sampling engine, though there have been cases where a language is "forked", and separate engines built for the same language (like WinBUGS and OpenBUGS).

This is not a simple explanation, but one I quite like (by Dan Roy) https://www.youtube.com/watch?v=TFXcVlKqPlM

It maps observations where you have a clue about how they were created (priors) to a stochastic function. So quite literally, you end up with a probabilistic program.

There are many solutions for each prior. Priors can for example be expressed as computational graphs with random variables.