Comment by srean | Hacker News Reader

Comment by srean | Hacker News Reader

Parent Full thread

At the core both derive their optimization from the distributive property. If the expression graph has symmetry, you get more optimization out of it.

https://www.cs.ubc.ca/~murphyk/Teaching/Papers/GDL.pdf

Check out the first paragraph

    THE humble distributive
    law, in its simplest form
    states that...this leads
    to a large family of fast
    algorithms, including 
    Viterbi’s algorithm and 
    the fast Fourier
    transform (FFT).

Two extremely influential papers appeared back to back in transactions information theory. This is one of them.

The other is

https://vision.unipv.it/IA2/Factor graphs and the sum-product algorithm.pdf

Both are absolute gems of papers. The editor made sure that both appear in the same volume.

Interesting, of course many computations can be expressed as a graph. In the case of the bipartite graph we perform belief propagation on to decode LDPC where is the optimization from the distributive property? The parity matrix would typically be constructed so that there's few subexpression to factor out, to maximize the error correcting properties.

I agree both FFT and belief propagation can be expressed as message passing algorithms.

It shows up in pushing in the parenthesis and pulling common terms out in the expression that is a sum (over all possible assignments) of products of terms.

Doing the summation the naive way will be exponential in the number of variables. The goal is to this in an efficient way exploiting the distributive property and symmetry if any, much like in the FFT case.

This can be done efficiently, for example, when the graph is a tree. (Even if it isn't, one can pretend as if it is. Surprisingly that often works very well but that's a different topic entirely)

Read the paper it's not difficult to follow.

The second link is broken on HN because it contains a space. Here's a clickable version: https://vision.unipv.it/IA2/Factor%20graphs%20and%20the%20su...

There’s a whole subfield called generalized distributive law https://en.wikipedia.org/wiki/Generalized_distributive_law

Reply on news.ycombinator.com