How to Explain Deep Learning Using Chaos and Complexity

Here is an interesting paper that pinpoints the influence of regularization on learning with Neural networks. From the paper:

> Our central finding can be summarized as:

Deep neural networks easily fit random labels.

and later:

> While simple to state, this observation has profound implications from a statistical learning perspective:

> 1. The effective capacity of neural networks is large enough for a brute-force memorization of the entire data set.

> 2. Even optimization on random labels remains easy. In fact, training time increases only by a small constant factor compared with training on the true labels.

> 3. Randomizing labels is solely a data transformation, leaving all other properties of the learning problem unchanged.

I subscribe to Schmidhuber's notion that algorithms that rely upon randomness to select parameters have a very illusory "simplicity" that relies upon externalizing the complexity of the PRNG. http://people.idsia.ch/~juergen/newai/