> One thing that we wanted to try but didn't have time was to learn a generalization of convolution. Since convnets are a special case of fully-connected nets with some weights tied and some set to zero, we were thinking of ways to parameterize this weight tying that included both fully-connected and convnets as a special case.

I see, that's cool. So you could define a 'symmetry operator' the defines a regularization term as e.g. k * sum_{ij}[(w_{ij} - w_{i+i',j+j'})^2] (modulo my hand-wavy tensor indexing). Then the dk tells you how much a layer wants to be a convolution and di', dj' tells you what convolution kernel size it wants to be.

(edit: made less badly articulated)