I try to highlight the paper and some of the history and relevance, in a general way, but also hitting the math hard.
the more general idea, which I am still formulating, is that DL systems are very different from traditional ML (ala VC theory)
In traditional ML systems, we tune the regularizer to optimize the capacity of the learner
In deep learning, we would optimize both the capacity (entropy) of the learner, and the optimization problem (energy function)
This is also what happens in the stat mech of protein folding, where the energy is optimized, even when we are at minimum capacity. This gives rise to a funneled energy landscape
https://charlesmartin14.wordpress.com/2015/03/25/why-does-de...
Similar behavior is seen generally in stat mech near a critical point, and this is why the RG analogy is relevant for me.
We should be able to see this behavior if we simply plot the entropy vs energy of say an RBM. Im not entirely sure yet how general this is, but it works for MNIST.
I discuss (some of) this in more detail in a video from a talk I gave this summer at MMDS https://www.youtube.com/watch?v=kIbKHIPbxiU