Hacker News Reader

Comment by amalcon | Hacker News Reader

The novel part is not that the shallow network can compute the same functions (that is interesting, but that particular result is decades old), but the notion that the shallow network can learn the same functions efficiently.

What we already knew: For any given neural network of N layers, there exists a 3-layer neural network computing the same function.

What we didn't know: There are (at least reasonably often) ways of training shallow networks that get similar efficiency to the ways of training deep networks given the same functions.

View on HN

No, they require training a deep neural network first. Then training the shallow net to mimic the deep one.

The claim is that the shallow net can have the same number of parameters as the deep net. It's always been known that a large enough shallow NN can theoretically approximate any function. But that it can do so with few parameters is very surprising.

My best guess is that the majority of parameters in deep NNs are unused or redundant. I.e they have a low weight, or don't influence the output very much, or another neuron computes mostly the same function. Many types of regularization like dropOut and weight decay heavily encourage this.

Whereas the shallow NN is forced the maximize the usefulness of every single parameter to get the same result. It doesn't have to worry about overfitting. I am not sure if the paper accounted for this though, it's been awhile since I read it.

Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?

If I understand correctly, they generated artificial data from the NN they were mimicking.

No, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters.

Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.

Their method for training a shallow network requires that one first train a deep network.

Reply on news.ycombinator.com