Probabilistic programming does in 50 lines of code what used to take thousands
newsoffice.mit.edu
newsoffice.mit.edu
Heck, I spent months on a binge reading functional programming research papers, and it still doesn't mean I know any functional languages other than very superficially.
For a great example look at all the papers Andrew McCallum's group has been able to publish by building on top of FACTORIE: they get to focus their time on the problem at hand rather than all the math required to solve it. Basically they can write code that generates a model dynamically but the framework handles all the inference. Compare that to how these things are built without such a framework: you spend most of your time painstakingly hand-deriving update rules and then implementing them as code.
IMHO, the exciting thing is that ML is getting closer and closer to being an everyday tool for engineers rather than something that requires you to be a full-time math person to use effectively.
I so wish! As a Sr Data Scientist, I interview potential candidates quite often, many of these are 10x engineers.
Me: (2,3,4) is a vector.
Eng: Ok.
Me: Gimme a unit vector in the same direction.
Eng 1: ???
Eng 2: "It can be done. I don't know how, but with Spark it can be done".
Eng 3: I will need R. ( given R, he fiddles with it for 5 minutes getting nowhere fast )
But come on, everyone knows how to use the definition of the standard (l2) vector norm to normalize a vector! Don't joke!
What little I remember about vectors is from my high school maths classes, and in my 20 years of doing software engineering it's come up exactly once (for a GIS related project).
I wouldn't expect anyone working with machine learning to not know these concepts either.
I don't think libraries count in terms of code. We all use code to program. Standing on the shoulder that preceded us. Using a library and a function should just count for the most part.
It's like the difference between a complete kitchen that fits in your pocket and an iPhone app that lets you order a burrito. The article suggests something like the former. A library which encapsulates 1000 lines of code into a single function call is like the latter.
On the other hand, there are things like Prolog. You can think of Prolog as a backtracking constraint-solving library, and then another library that parses a DSL for expressing facts and procedural constraints and feeds it to the first library. But Prolog's language isn't really a DSL, because it isn't particular to any domain: there's no closed solution-space where Prolog applies. The efficiency gains you get from Prolog's elision of proceduralized contraint-solution code can apply to any program you write. And so its value is unbounded; its ROI is certainly positive, whatever the cost was to implement it.
That's the comparison that's useful here, I think. Is this something that only solves problems in one domain? Or is this something that could be applied to (at least some little bits of) any problem you encounter?
Python also has some mandatory libraries if you want to do any specific. Numpy and pandas for statistical analysis make them required.
My code is concise and clean but it is because of these libraries.
I am also certain that there are exceptions.
.MODEL TINY
.CODECan someone knowledgeable in graphics research explain the context that this question comes from?
If I am reading the question correctly, I infer that the question suggests that there exists a right way to reproduce the visual experience of reality. To me, this sounds like a question that is equally valid to have no answer (or many answers) in aesthetics, art, and philosophy, etc.
Its Plato's Allegory of the Cave all the way down.
Imagine "watching" a movie compressed using your very own prior knowledge. Every scene could be described in couple of hundred lines of plaintext. Today we do this by reading a book :) What if we could build an algorithm able to render movies from books?
Bob Coyne has been working on a system for generating images of still scenes from text descriptions for about 15 years now:
https://www.wordseye.com/ http://www.cs.columbia.edu/~coyne/papers/wordseye_siggraph.p...
"The world is such and such or so and so, only because we talk to ourselves about its being such and such and so and so..." Carlos Castaneda
For a long time statisticians wrangled over this word in a reduced context. The "art" of statistics is to build a model of the world which is sufficiently detailed to capture interesting data but not so detailed to make it difficult to interpret as a human decision-maker. Statisticians usually solve this problem by building a lot of models, getting lucky, presenting things to people and seeing what sticks.
For a long time this lack of a notion of "rightness" was so powerful that it precluded advancement of the field in certain ways.
With the advent of computers we discovered a new, even more precise form of "right" however and this formed the bedrock of Machine Learning. The "right" ML is concerned with is predictive power. A model is "right" when it leads to a training and prediction algorithm which is "probably, approximately correct", e.g. you can feed real data in and end up with something useful (with a high degree of probability).
So with respect to computer vision we know that it is very difficult to build "efficient" algorithms, ones which work well while using a reasonable amount of training data. CV moved forward when it realized that there were representations of the visual field which led to better predictive power---these were originally generated by studying the visual center of human and animal brains, but more recently have been generated "naively" by computers.
So, there's a reasonably well-defined way that we can find the "right" representation of visual scenes: if we find one which ultimately is best-in-class of all representations for any choice of ML task then it's "right".
So in some sense optimal compression gives the best you could hope, up to limitations of the probabilistic models, which is why I like this explanation.
For example, if you can extract a 'Mesh' from a 2D picture, you can generate many other view points, and that mesh can be considered a good representation. If you are more sophisticated however (and perhaps have a larger "dictionary"), you can instead extract 'There are two wooden chairs 1m from each other, ...'.
That's the sense in which the representation is fundamental to computer vision -- it distills what the system knows (or what it wants to know) about scenes. The more concise the representation without loss of information the smarter your system is (and past a point becomes a general AI problem).
See for example:
Natural Image Statistics — A probabilistic approach to early computational vision https://www.cs.helsinki.fi/u/ahyvarin/natimgsx/
However, at this point, unless you design your program in a very specific way and use a lot of tricks, your sampler is very unlikely to converge, and you won't get any meaningful result without a gargantuan amount of computing power.
Reading probmods.org and dippl.org made me go from being very pessimistic I would see it in my lifetime to a solid maybe.
Let's take a step back and consider a logical problem. Can you put N socks in N-1 boxes such that no box contains more than one socks? Obviously not, it's the pigeonhole principle.
Convert that question into a Boolean circuit, and throw a SAT solver at it. It will die. In fact, using only first order logic, a proof of the pigeonhole principle requires an exponential number of terms. Looking at logical propositions alone is too myopic to solve the problem, you have to formulate higher level theories about the structure of the problem to solve it (in this case, natural numbers).
The same goes for probabilistic programming. As long as the paradigm is to treat the problem as a black box energy function to sample from, it is doomed to be inefficient. Try writing a simple HMM and the system will choke, even though there are efficient algorithms to sample from such a model.
If you look at deep learning techniques, they take an interesting approach which is to learn to approximate the generative distribution and the inference distribution at the same time. This is the basis of the work around autoencoders, deep belief networks, and it guarantees that you can tractably sample from your latent representation.
For example, I have been toying with using MCMH similarly to how it is used in dippl.org to write a kind of probabilistic program that generates other programs by sampling from a probabilistic programming language grammar. A bit like with the arithmetic generator here: https://probmods.org/learning-as-conditional-inference.html#... but for whole programs.
After the MCMH has converged to a program that generates approximately correct output, you can tune grammar hyperparameters on the higher likelihood output program samples so that next time it will converge faster.
I don't know if this counts as approximating "the generative distribution and the inference distribution at the same time" under your definition but my hope is that the learned grammar rules are good abstractions for hierarchical generators of learned concepts.
Of course the worry is that my approach will not converge very fast but there are reasons to think that having a suitably arranged hyperparametrized probabilistic grammar might use the Occams' razor inherent in bayesian methods to produce exponentially fewer, simpler grammar rules that generate approximations when it doesn't have enough data to converge to more complex and precise programs and that these simple rules which rely on fewer parameters might provide the necessary intermediate steps to then pivot to more complex rules. These smaller steps help MCMH to find a convergence path. Not sure how well it will work for complex problems however. My idea is still half baked. I have ton's of loose ends and details I have not figured out, some of which I might not even have a solution, as well as little time to work on this (this is just a hobby for me).
Anyways, all that to say that probabilistic programming can go beyond just hardcoding a generative model and running it.
How do you sample from the set of programs that produce the correct (or approximately correct) output?
You could use rejection sampling, but that would take very long as only a tiny fraction of all possible programs produce the desired output.
You could use a MCMC method, but the problem here is designing a proposal density. There is no reason to expect programs which are almost correct to be syntactically close to programs which aren't. Changing a single bit in a program tends to radically alter its behavior, in a very non linear way. Program space does not map smoothly to output space.
You mention hyperparameters, but what would those be? If you're building a probabilistic grammar, they might be things like emissions probabilities... but tweaking the proportion of "if" statements you produce, as opposed to, say, function calls, is hardly going to make you better at sampling from the set of correct programs.
In general, the idea of meta-learning is a powerful one, but without a way to guide the search, I don't see it as feasible.
You correctly identify the main hurdle. I admit I am not sure it will work.
However, I think it might be possible to design the programming language such that changing a few bits doesn't usually radically alter the output.
For example, let's say you are trying to learn a grammar to draw a blue irregular polygon. If there are primitives with fewer parameters that can get you an approximate output, say a circle, this makes possible a learning path that looks like "circle"->"blue circle"->"simple blue polygon"->"complex blue polygon". In addition to that, if the grammar rules that generate similar output can be clustered in the space of grammar rules, small jumps in this space may give you small output changes. Using bayesian priors will naturally use the simpler, fewer parameter shapes first and pivot to more complex ones as enough information is learned while, I think, creating these more complex rules close to the simple ones in the space of grammar rules. That is my hope anyways. I got it working as expected-ish with a simple vision example like I just described.
Ahaha, "this sounds like a job for topology!"
>The problem is that you need to sample efficiently from the posterior distribution, and for anything AI related, MCMC isn't going to cut it.
Which is precisely why so much of the actual research work in probabilistic programming is to add more and faster inference strategies. One paper already got a 600x speedup just by using some techniques from the compilers/VM world related to caching whole and partial computation traces.
Very probabilistic..
Hardly "distruptive". You should really think of them as DSLs that people actually use.
If you're intrested in such things, perhaps you should take a look at PRISM, which embeds a probabilistic framework within B-Prolog. http://rjida.meijo-u.ac.jp/prism/
For example, Stan-language supports MCMC modelling using Hamiltonian dynamics.
What's more interesting is that we're seeing heightened interest in these techniques again after they were ostensibly sidetracked in favor of statistical methods.
I'm not buying in just yet, because although it's nice to talk about model specification as completely independent processes, the availability of fast inference algorithms sometimes dictates what models you should choose. Sometimes less exact models with a larger parameter space that allows you to crunch orders of magnitude larger datasets (with approximate inference algorithms) yield more useful results than better specified models...and sometimes not. The thing if one still needs to know the whens and whys of picking certain models over others, and can't just gloss over the inference details.
But, this (and Church) look very interesting.
For example, I refer you to the paper wherein some biologists "discovered" integrals in 1993. http://care.diabetesjournals.org/content/17/2/152.abstract
Good job!