Hacker News Reader

The Generalization Gap (test/train) seems to me to be quite unsatisfying as a definition, and then an appeal that theory and practice are different doesn't make me feel that anything is explained.

Being able to create new regularisation techniques is impressive though... looks like something that will need a read!

View on HN

I haven't read the paper yet, but when people say things like "We don't know why neural networks work", what they mean is we don't understand why they generalise so we'll, by most prior theory they are quite over parameterized, so while they should be able to fit the data - and it has been shown that typical architectures can fit random data - previous theory said the penalty for this is that it will not generalize well to new data from the same distribution.

So generalization bounds on NNs are actually the key thing that people want from theory.

I think the reasoning is that optimizing the training performance is "easy", whereas optimizing the test performance is "hard". If you can guarantee that test performance will be close to training performance, then optimizing the test performance becomes "easy".

Reply on news.ycombinator.com