Hacker News Reader

Comment by x0x0 | Hacker News Reader

Ease of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand quite a bit of math, where gibbs or other samplers like those in bugs/openbugs/jags/stan can work with the factored distributions and require much less mathematical sophistication from users.

View on HN

I know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days.

Perhaps a more important difference is that MCMC, while slow, is exact in the limit. Variational methods won't converge to the true posterior no matter how long you run them. You'll converge to an approximate answer which depends on the particular variational form you choose to use.

Are you aware of any work (or researchers) working on that? I would be very interested.

And I think your second sentence reinforces my point -- it makes variational methods either more fiddly, or require more understanding to use well.

edit: here's one such (limited but nice) effort: http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pd...

Microsoft's Infer.NET is essentially an automated system for variational (/expectation-propagation) inference. It implements primitives for common operations and distributions, and then uses the local structure of mean-field inference ("variational message passing": http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf) to build up variational inference algorithms for arbitrary factor graphs. It's not infinitely flexible and doesn't solve all problems related to variational inference, but once you get used to it you can iterate quite quickly on model refinements, inference tweaks, etc. without any tedious derivations.

thank you!

I think we are broadly in agreement. When I say coding up inference for a particular model, that includes deriving the equations and updates needed. This effort is nonzero for all inference methods, but is much lower for MCMC.

You're totally right; I should have read it that way.

Reply on news.ycombinator.com