Correct me if I'm wrong, conceptually, probabilistic programming does to equation solvers what declarative language like SQL do to query data.
Say you have a model, y = mx + b, and you have some data. Normally you can solve this explicitly with matrices or whatever. But you'll need to implement the solution. With probabilistic language like BUGS, you can feed in the model and the data and it would return `m` and `b` as a probability distribution.
This is an trivial example. One example of where it really comes in handy is doing bayesian inference.