Why does unsupervised deep learning work?
arxiv.org
arxiv.org
+ An autoencoder is a stabilizer of the input f: it maps it to itself.
+ Imagine that the space of autoencoders forms a group.
+ Learning stops as soon as a stabilizer is found.
+ If the search is a Markov chain, the bigger the stabilizer the sooner it will be hit.
+ The group structure implies that this big stabilizer corresponds to a small orbit.
+ The orbit of an element x in X is the set of elements x can be moved to by the elements of G (the group).
+ The stabilizer is the set of group actions that map x to itself.
+ Blog post explaining fundamentals: https://gowers.wordpress.com/2011/11/09/group-actions-ii-the...
+ Reconstruction in deep NNs is often guided by an l2 distance. If there are competing feature sets, gradient descent moves the configuration to one of the stabilizers.
- Note. Is this indeed the case? Is there a WTA mechanism at play? Is this preferable?
- Note. In this view a NN is a MCMC that is stopped prematurely.
+ The probability that a network "discovers" a stabilizer for the signals f_i depends on the volume of the stabilizer.
- Would we be able to construct a PDF by running an MCMC till we are in a high probability region? Or does this only work if we're interested in modes?
+ A neural network operation is not a group action but can be seen as a "shadow group" in some auxiliary space (theorem 4.1, 4.2).
+ The size of the stabilizers is preserved in the original space (theorem 4.3).
+ Next levels exhibit again symmetry, generalized edges lead to figures like trapezoids, triangles, and butterflies.
Most important take-away: representations that exhibit a lot of symmetries are the ones that are easiest/fastest to find.
Generally the 'performance' of the autoencoder is measured at the reconstruction step, i.e. what is the reconstruction accuracy of the autoencoder? This is an L2 loss function.
There have been other links listed on HN in the past. One that came back a few times is on renormalization group theory (actually a semigroup).
Currently my new angle of attack is to check what people have to say who come from a representation theory angle.
I also liked this paper by Persi Diaconis a lot: http://statweb.stanford.edu/~cgates/PERSI/papers/The%20MCMC%...
I also like that on his wikipedia page (https://www.wikiwand.com/en/Persi_Diaconis) it's stated that: According to Martin Gardner, at school, Diaconis supported himself by playing poker on ships between New York and South America. Gardner recalls that Diaconis had "fantastic second deal and bottom deal". :-)