I strongly agree with this paper. Regularizing a NN is the key for better performance. But first, one needs to know, what exactly to regularize. I don't think trial-and-error hyperparameter bingo should be the way to go. We need better insight and understanding about these networks, analyze their structure and find out, what exactly is wrong with it. Then, a TARGETED regularization (layerwise, or maybe per neuron) has a huge potential to let very simple networks perform extremely well. I even suggest, that adaptive regularizations (on/off/strength) should be researched even more. It is not necessary, that a network is regularized all the time the same way.