The Algorithms Behind Probabilistic Programming
blog.fastforwardlabs.com
blog.fastforwardlabs.com
> The distribution must be smooth (this method doesn’t work for discrete parameters), but it need not be analytically differentiable
'Smooth' and 'analytic' both have basic technical meanings in the context of differentiability [1], [2].
Glossing over the assumptions in this way is quite misleading
[1] https://en.m.wikipedia.org/wiki/Smoothness [2] https://en.m.wikipedia.org/wiki/Analytic_function
The algorithms the article define seem to justify seeing probabilistic programming as an advance beyond twenty line programs that call 20K sized libraries - basically, these algorithm inherently auto-tune a lot of higher level operations so these can be less-leaky abstractions.
http://probcomp.csail.mit.edu/bayesdb/
Almost always what I want is closer to SQL-like questions that I later want to manipulate with languages like Ruby or Python, rather than having to load all the data into a giant array and then have to use probabilistic programming on the whole damn thing.
Probabilistic programming is an area of huge academic activity, with deep connections to statistical inference, compiler design and, yes, UX.
Remember the origin of the term "dynamic programming"?
> "I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, Where did the name, dynamic programming, come from? The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word “programming”. I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let's kill two birds with one stone. Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities."
Reminds me of myself, early in my career. "Well, it probably works..."
Another way to look at it is, similar to how you can have the concept of a program as a differentiable function, you can have probabilistic programs as defining a distribution. Their utility is the ease with which they allow you to model complex probability distributions and then condition on observations. With that, you can do inference.
If you've heard of bayesian networks or graphical models, PPs can generalize these. It's just a few lines to do hidden markov models, bayesian regression, gaussian processes, gaussian mixture models and LDA topic models. Their downside is that performing inference has been less efficient than writing these models out by hand. We know that the ideas are powerful as they produce generative models which work off less data and naturally incorporate uncertainty.
The dream is, just as a more expressive language should have less patterns, probabilistic programming will one day make writing complex generative models so routine people won't bother specially naming them as this or that machine learning algorithm.
^ The kind you find in a book like this: http://www.mbmlbook.com/toc.html
PP itself is not just an API for stat functions and data. It lets you write a generative, forward program, and then you run the program on outputs (data), and it generates probable inputs (parameters). It's similar to constraint programming paradigm, but based on probabilities.
No, you're just dismissing it without understanding it first. Which seems super common on this forum.
Kerning is also messed up, the distance between the letters is too large in comparison to the distance between the words.