I mean yes this is neat, but inference engines already work declaratively with factor-graphs and potentials.
Are there techniques exploiting algebra/logic that helps make inference faster in practice ?
Deep learning may have the edge currently for being a bit more mathematically tractable and much easier to massively parallelize, but this seems to me like a more fundamental foundation for AI (there remain issues to be solved with it though).
These languages are used to describe models and run probabilistic simulations of them but can also be used to describe other programming languages probabilisticly. This means the potential to go up one leve of abstraction to a probabilistic program that writes other programs to model and predict the world.
Here's a description of one of my failed attempts at this:
https://www.quora.com/What-deep-learning-ideas-have-you-trie...
sampling is slow.
general AI needs structured prediction. yes, graphical models can do that (HMM, CRF etc.) but inference starts to get slow and special case implementations are required for different domains. [1]
I see no way for someone getting automatic inference and training for [1] with a probabilistic programming language.
given the new deepmind paper on discovering shortest path algorithms, it's quite clear that structured predicition assisted by deep networks works quite well (this was demonstrated by a vast array of work) and graphical models represented by probabilistic programming languages are far away from being that successful.
>sampling is slow.
Right. Sampling is slow. That's why automated variational inference is a very active field of research these days: instead of approximating the posterior by sampling, you approximate it with an optimization problem whose gradient-descent provides a bound on the posterior probabilities across the parameter space.
All the work on training deep neural networks has made our hardware and software very efficient at solving optimization problems.
Only based on that, this will be useful for anything regular programming is useful.
In addition, this makes modeling uncertainty much easier (in the sense of "closer at hand"). That may allow for new ways of dealing with user input. Instead of saying "is this email address valid or invalid", we can start asking questions like "Is the probability of this email being valid larger than 99%? Then we'll accept it. Is it larger than 95%? Then we'll ask the user to confirm it. Is it less than 95%? Then we'll tell the user it's incorrect and have them retype it."
These are things we normally don't care to model because it would require lots of additional machinery. With that machinery built into the programming language, it is much easier to reach for, potentially with a better user experience to boot.
I was trying to find out what more it can do other than parsing a DSL into a graphical model.
The probability that the user switched off Wifi on their machine is 4%, and the probability that their router is having trouble is 3%. These numbers can be based on actual measurements. What do we tell the user when they have problems connecting? Instead of just saying "it's either this or that" we can run the numbers, and perhaps in aggregate there's an overwhelming probability it's a particular event, in which case we suggest that first.
There are so many cases where we don't actually know for certain all the parameters involved, but the conventional approach is still to round the probability either up to 100% or down to 0%. Simply because that's easier in conventional programming languages. As a result, you might not see these events as having probability distributions, but they do.
Maybe you're not impressed with graphical models due to your experience dealing with them according to the points above.
by "writing code that generates a sample from the joint distribution", you can achieve much better modelling and control than what it would take with the normal methods of producing graphical models.
I see it as having a database of facts about the world with attached probabilities that tell you which view over (or perhaps version of) the world is the most likely.
And then you can do EM search for optimal parameters. Learning, right? It's all built-in to the language and you don't need to hand-craft task-specific versions depending on your domain (like Baum-Welch, Inside-Outside etc).
Also, it's a probabilistic Prolog: it's Turing complete and gives you all the expressive power of first-order predicate calculus. With probabilities. And learning of parameters from data.
Languages like this go way, way beyond ad-hoc implementations of inference over graphical models, to giving us a new vocabulary to express reasoning over vast sets of data.
__________
>> However, many of the most innovative and useful probabilistic models published by the AI, machine learning, and statistics community far outstrip the representational capacity of graphical models and associated inference techniques.
>> PROBABILISTIC PROGRAMMING LANGUAGES aim to close this representational gap, unifying general purpose programming with probabilistic modeling;
I see it as giving the tools to the community to describe their models and automate inference over them in a unified manner that can be communicated more easily, and in a way that is better understood by all.
The research listed in the page (http://probabilistic-programming.org/research/) appears to take the following courses,
Stochastic processes-ish,
- Inference techniques for handling recursion.
PGM-related work,
- Parallelization
- Optimizations for MCMC based on structure
(Old school) Theoretical CS,
- Formalisms reminiscent of Languages.
It's still not entirely clear to me how important this work is; though heavy weights like J. Tenenbaum and others continue to work on it.
The page says that many models can't be subsumed under PGMs, and yes that is true for things like PPCGs and other recursive things (martingales, stochastic processes..).
However, things that PPLs are known for like the inverse graphics work, are really PGMs. It's entirely possible that what I'm asking is akin to questioning the significance of the Deep Neural networks and assorted frameworks, in contrast to chain rule; but considering that it is more than getting X% at Imagenet here, I think it is a reasonable question to ask. Is it about the representation or the implementation ?
- Inference on loops seems to be something that can be handled as well (dynamically ?).