The Design and Implementation of Probabilistic Programming Languages | Hacker News Reader

The Design and Implementation of Probabilistic Programming Languages

dippl.org

It seems like quite a hot research area, and the quality of projects being released is pretty impressive. Microsoft Research's work in this area is pretty good too (Infer.NET and Tabluar)

http://research.microsoft.com/en-us/projects/fun/

Interesting how they now reverted back from a Scheme-like language, previously also published by Goodman at https://probmods.org/

Do you think it's because of all the parentheses that people can't understand?

They use a mostly functional subset of javascript (You are not allowed loops). It is much more readable than scheme if you ask me but not necessarily more powerful. You get the benefits of the very powerful and ubiquitous javascript compilers that exist nowadays which I like.

I played with it and you are somewhat limited right now. The first issues is that you quickly hit the top of the javascript stack because of the recursive style (remember: no loops). The most elegant solution to that would be for browsers to support tail call optimisations which is in the ES6 standard (Are you listening Mozilla or Google? If you implement this part of the spec sooner you could be leading the way for the most promising area of AI and maybe of computing in general). Although there are possible work around namely if the webppl compiler were to use trampolines or "promise" like structures. IMO, it would clutter the code and be much less elegant than native ES6 TCO.

The second limitation is that you are not allowed mutability. There is a git branch with support for mutability partly implemented. I'm not sure they should go that way though. I've never really been into functional programming style but in the case of probabilistic languages, I really see the value. It's much simpler and easier to follow what is going on when debugging. Supporting mutability also makes the compiler more complex and memory usage high as you have to copy the whole memory footprint of your program at each probabilistic sample you take. It is not that difficult to instead include something like mori http://swannodette.github.io/mori/ to manipulate arrays immutably.

This is the next breakthrough coming up after Lisp...

Probabilistic programming is already basically the definitive Next Big Thing in machine learning, since it allows us to automate reasoning over far more sophisticated models than we could previously write down by hand, while the approximate inference techniques being used make numerical evaluation of Bayesian reasoning tractable.

The generality and power of the approach are, frankly, amazing.

MCMC probabilistic programming won't be the next big thing in machine learning though. You need some sort of maximum likelihood / maximum a posteriori / variational bayes to do machine learning. MCMC is far too slow.

Maybe not. Not all probabilistic programming relies on MCMC, you can do MAP with probabilistic programming too. But in general, it's true that it tends towards Bayesian integration.

In a sense, MCMC tries to solve #P complete problems, while MAP "only" tries to solve NP complete problems. The former are probably intrinsically harder than the latter.

However, it's quite possible that some clever implementation might yield results which are as good as MAP in the same amount of time, and progressively refine them with more time.

There are two free lunches with the curse of dimensionality: - the manifold hypothesis - Monte-Carlo integration

True, the dimension rears its head as an ugly multiplicative constant in the latter - still, there has to be something there.

Many of these PPLs use variational Bayes.

At least according to my reading, fast algorithms for calculating expectations and modes via sampling and approximation actually came before the ones for doing generalized probabilistic inference via sampling and approximation.

Yes, but it's not widely commercial yet. Huge startup opportunities! We are thinking about it. If wanna get in touch find my contact via http://akuz.me (my personal website)

I want to build a probabilistic operating system. People don't understand what I mean when I say that. Argh.

What do you mean, a probabilistic operating system?

Probably some kind of operating system that relies on probabilistic principles.

I hope it's not an operating system that decides to run your thread based on the output of a random variable. And also I hope this is not a realtime operating system. That would probabilistically speaking be the lower depths of hell.

Reply on news.ycombinator.com