Hmmm - I'm also thinking that this is one of those things that probably has a much better explanation - and the science/maths will (hopefully) backfill why it works so well.

I half-remember from somewhere that as long as the gradient descent direction has the correct sign 'in expectation', then the SGD will ~work. So there's a whole lot of flexibility in there for having a good idea that at least doesn't fail horribly.

For instance, in other DeepMind work, they do lots of asynchronous weight updates - and the accuracy decrease from ignoring any kind of 'locking' is dwarfed by the speed increase of being able to run more stuff in parallel.

Another image I can't shrug off is that of Q-learning in a game, where the updates implicitly pass back from 'the future' (which also works ~better than it should). In this case, the linear model would just be an estimator of where the update values are going to land...

It will be an approximation of the Jacobian because it is just backprop. It also seems they only use the linear model when they also condition on the correct class which makes things easier. It is also only MNIST (10 classes).