duvenaud

Right, once we start heavily hyper-parameterizing our models, there's the potential to overfit the hyperparameters. I think most people agree that has already happened on common datasets such as MNIST.

However, the current method of avoiding hyperparameters is just to have very few of them. This is kind of like avoiding overfitting of parameters by only having 10 parameters in your model - barbaric!

That being said, for hyper-gradients to really be useful, someone needs to develop hyperparameter optimization schemes such as BayesOpt that can condition on gradient information, to allow us to try different hyperparameter settings in parallel. I know at least one group is working on this, but as far as I know it's not ready for prime time yet.

Exactly. The problem is that it's hard to find a good balance between having too many hyperparameters, versus basically telling it the answer beforehand. In particular, it was hard to figure out how to let it learn the stride of the convolution, but perhaps we could have done without it.

The other problem is that to get a reasonably-sized convnet by pruning and tying a fully-connected net, you need to start with a huge fully-connected net, which slows things down.

Thanks! Yes, great suggestion. One could also use a similar trick to learn the number of layers, by regularizing unused layers to be close to the identity function.

One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case.

Yes, gradient-based hyperparameter optimization is one option for taking the human out of the loop to some extent. Of course, we can't avoid having to specify hyper-hyper-parameters!

More generally, automatic hyperparameter tuning (without gradients) is already a standard part of many researchers' pipelines.

It's also part of good scientific practice, since it makes it harder to bias the results towards a particular method.

Authors here, feel free to ask us anything!