Authors here, feel free to ask us anything!
From what I've followed of machine learning, I feel like there are areas that are not very rigorous, for example where humans have to tune weights and parameters. Does your work address those places?
Specifically do you think you can train on enough parameters, and compare enough results, to begin to estimate how that tuning can be done automatically?
More generally, automatic hyperparameter tuning (without gradients) is already a standard part of many researchers' pipelines.
It's also part of good scientific practice, since it makes it harder to bias the results towards a particular method.
You kind of allude to things like this with the "discrete parameters" comment, but here's how I imagine one could use this to meta-optimize layer sizes, which are after all continuousish: make all layers too large, then compute the hyper-derivative dn_i of a per-layer cutoff that starts regularizing units more and more strongly beyond some unit index n_i for each layer i (as in the regularization strength is sigmoidal, or better exponential, centered around n_i). Does that make sense?
Also, I immediately wanted to see vector gradient field plots in hyperparameter space of a bunch of toy models. It'd be fun to actually see "hyperconvexity".
One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case.
I see, that's cool. So you could define a 'symmetry operator' the defines a regularization term as e.g. k * sum_{ij}[(w_{ij} - w_{i+i',j+j'})^2] (modulo my hand-wavy tensor indexing). Then the dk tells you how much a layer wants to be a convolution and di', dj' tells you what convolution kernel size it wants to be.
(edit: made less badly articulated)
The other problem is that to get a reasonably-sized convnet by pruning and tying a fully-connected net, you need to start with a huge fully-connected net, which slows things down.
Yeah, it seems impossible because changing the stride changes the size of the output tensor completely.