Hacker News Reader

Comment by BenoitEssiambre | Hacker News Reader

In my humble opinion something like this will be at the root of the solution to general AI.

Deep learning may have the edge currently for being a bit more mathematically tractable and much easier to massively parallelize, but this seems to me like a more fundamental foundation for AI (there remain issues to be solved with it though).

These languages are used to describe models and run probabilistic simulations of them but can also be used to describe other programming languages probabilisticly. This means the potential to go up one leve of abstraction to a probabilistic program that writes other programs to model and predict the world.

Here's a description of one of my failed attempts at this:

https://www.quora.com/What-deep-learning-ideas-have-you-trie...

View on HN

this stuff existed for decades and it hit a wall.

sampling is slow.

general AI needs structured prediction. yes, graphical models can do that (HMM, CRF etc.) but inference starts to get slow and special case implementations are required for different domains. [1]

I see no way for someone getting automatic inference and training for [1] with a probabilistic programming language.

given the new deepmind paper on discovering shortest path algorithms, it's quite clear that structured predicition assisted by deep networks works quite well (this was demonstrated by a vast array of work) and graphical models represented by probabilistic programming languages are far away from being that successful.

[1]: http://www.philkr.net/home/densecrf

>this stuff existed for decades and it hit a wall.

>sampling is slow.

Right. Sampling is slow. That's why automated variational inference is a very active field of research these days: instead of approximating the posterior by sampling, you approximate it with an optimization problem whose gradient-descent provides a bound on the posterior probabilities across the parameter space.

All the work on training deep neural networks has made our hardware and software very efficient at solving optimization problems.

I don't think it has to be either or. For example, you can implement a Deep Net in a probabilistic programming framework (http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learn...) like PyMC3. The inference here is not using sampling but rather ADVI (http://pymc-devs.github.io/pymc3/api.html#advi) which is almost as general but much much faster, and can be run on sub-samples of the data (similar to stochastic gradient descent used in deep learning). Once we bridge these two domains we can get the best of both worlds, like a deep net HMM.

Reply on news.ycombinator.com