Generalization in Deep Learning [pdf]

I think the reasoning is that optimizing the training performance is "easy", whereas optimizing the test performance is "hard". If you can guarantee that test performance will be close to training performance, then optimizing the test performance becomes "easy".

I haven't read the paper yet, but when people say things like "We don't know why neural networks work", what they mean is we don't understand why they generalise so we'll, by most prior theory they are quite over parameterized, so while they should be able to fit the data - and it has been shown that typical architectures can fit random data - previous theory said the penalty for this is that it will not generalize well to new data from the same distribution.

So generalization bounds on NNs are actually the key thing that people want from theory.

I use a customized version of this: http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/p...

The most important thing for me is to not read it from the beginning to the end (which is hard for me). Abstract->Conclusions, scan headlines. Methods is the most curious section for me. Depending on the paper and what I am working on I read the methods section last or first (even before the abstract). If it's more of a "oh that seems neat" paper I skip the methods section and mostly extract the idea. The book "How to read a book" is also a good source of ideas.

I don't think there's a one size fits all approach. I also find the various papers on writing literature reviews very helpful (for gathering an overview of a topic). Just checked my Zotero and these are the ones I have tagged:

"Using grounded theory as a method for rigorously reviewing literature"

"On being ‘systematic’in literature reviews in IS"

"A hermeneutic approach for conducting literature reviews and literature searches"

"Systematic literature reviews in software engineering–a systematic literature review"

"Writing narrative literature reviews."

Feel free to go as meta as you want ;)

read the abstract, read the conclusion, try to work out what the authors are trying to say.. hunt for the pay off in the paper, see if I can understand that, look for the results and check to see if I can understand that what they claim to show supports the pay off then start reading it. The actual reading can take up to six weeks for a big journal paper.

True.

> how long does it take you to read one paper?

It really depends. A deep learning paper just presenting a new architecture can be read in a few hours, meanwhile for some heavy papers on the mathematical side can take days, weeks to fully digest.

In the end it doesn't matter, you can be as slow as you want :)

> how long does it take you to read one paper?

It depends on to what extent the paper is self-contained.

Proof-heavy papers can take days or weeks to understand. It takes multiple reads, working through the math on scrap paper, and frequently discussing proofs with colleagues.

My process is generally to read through the paper a couple of times and then start tackling the proofs (if there are any). In general, I find it best to start with a very high level understanding and slowly descend into a greater detailed understanding.

It is a problem that is well-known and easy to solve. Their point is to show that they offer a boost in terms of training speed.

The empirical results aren't really the "point" of this paper.