Imitating Deep Learning Dynamics via Stochastic Differential Equations

Maybe pdf graphics would cost less? Keeping it scalable would be great

You should keep the vectored graphics for research papers, it is really handy so people can zoom in. The issue here is just how much information is in the plots you have. E.g. Figure 3 each subplot contains a lot more information than is actually needed. You could use moving averages there to clearly state the same thing. Or if you wanted to, you could include the envelopes. That would create smaller sized images. The other option is to just make the plots more sparse. There's far more information than necessary there.

It's weird, but images often sell a paper, so it is worth time learning how to make good images (or at least one person in your group). But that said, research comes first and that is more important. Just don't underestimate the power of good images. I often find this video on colors helpful[0]

[0] https://www.youtube.com/watch?v=Qj1FK8n7WgY

Vector graphics are much more complicated than raster images. They have to be drawn sequentially (to not mess up draw and blending order), mostly on CPU with the current software stack. And no, at 30MB you are looking at about 100000 paths, each of which consists of bezier curves which needs to be converted into polylines before drawing (there are algorithms that handles curves directly, but they are just as expensive). It's a vast amount of workload.

Images are relatively well handled with GPUs, and with a precomputed mipmap they can be rescaled very quickly, unlike vector graphics which needs to be re-rendered each time zoom level is changed.

> does it have direct application for training nets faster?

That's a great question! Unfortunately not yet -- though we believe further studies may bring us there finally. We found (at least for simple classification tasks) the features seem to have a two-stage behavior: a de-randomization stage to identify the best direction in the feature space; and an amplification stage where features stretches along these directions.

We've been thinking to identify a bound on the exit time of the first stage, and examine how it depends on different hyper-parameters, dataset properties etc, so that one may pinpoint how to reduce the time spending in the first stage, effectively making training faster.

> i.e. can the ode be integrated faster than backprop?

Also a good question, at this stage we need to estimate model parameters (the drift matrix) from simulations on DNNs. As future works we hope to explore if we can pre-determine those parameters so a comparison between backprop might make more sense.

With regard to the class conditioned regime they are experimenting with; this is merely attempting to explain more precisely _how_ deep nets are able to distinguish features between classes. We already know that they do; but we lack a detailed model of precisely why they do and my understanding is that many base assumptions made by e.g. statistics will not help you at all with neural networks (for instance, overparameterization leading to better performance on out-of-corpus rather than overfitting).

> I thought the main selling point of deep learning that it finds non-linear connections in the data.

Agreed. Yet there are unknown feature spaces in which the deep nets separate the data linearly; the mapping learnt by the deep nets from the data to this space is likely highly nonlinear.

For example, a binary logistic regression can be thought as mapping data to a 1D space ([0, 1]), and separates the two classes at say 1/2 linearly.

Thanks for sharing the interesting IRM paper!

Will read and be back for discussion hopefully soon.