Hacker News Reader

Comment by ricksharp | Hacker News Reader

Could someone list some practical examples where Differential Programming would be useful?

I am familiar where Nueral Networks and Convolutional Networks have done well especially around image processing etc.

But I can’t imagine where having differential code would help unless it is just tying multiple neural networks together in a continuous chain of differentiation.

For most programming tasks, I can’t imagine how differentiation would be possible or beneficial.

Is there a possibility that one could start with a series of unit tests and partial results and through gradient descent actually arrive at additional passing test cases? Most of the time in my experience, passing additional test cases like this requires significantly more complex structures that would not be found via differentiation.

View on HN

There are examples quite far from neural networks, the ones I can think of are broadly optimisation problems:

Many physics problems involve trying to find a function which minimises something -- energy, entropy, action. Or the state which makes the difference between two things zero. Sometimes adjusting many parameters slowly down the gradient is a good way to find these.

In bayesian statistics, the basic problem is to sample from a distribution, which you know only indirectly, by some kind of monte-carlo method. But the space to sample can be enormous. If I understand right, advanced ways of doing this exploit the gradients (of functions defining the distribution) to try to choose samples efficiently.

People have hacked tensorflow to do all sorts of things which its creators didn't intend. Or written tools specialised for another particular domain (like Stan). I guess the excitement is that instead of re-inventing the wheel in each domain, maybe this can be pushed down to become a language feature which everyone above uses.

For instance in robotics, the inverse kinematics could be formulated using the well-known equations, and combined with a CNN for vision, maybe do reinforcement learning across the whole thing. Or in audio classification, could use standard filterbanks construction for creating a time-frequency (spectrogram) representation, then combined with a CNN for classification.

Right now these things are done either: 1) Unstructured end-2-end learning, where a deep neural network has to discover the laws of physics unaided by structure. The models are massively over-parametrized, require a lot of data to train and vulnerable to adverse inputs. 2) As independent systems, optimized separately. If the first model is complex, the model that follows must usually reflect this complexity. A simpler global solution might exist, but cannot be found.

It is however super early days. Right now there are hints that going in this direction might be fruitful, but as far as I know, not many concrete wins or a lot of practice. That will take some years still.

In quantitative finance automatic differentiation will give you sensitivities (which tell you how the value changes when interest rates and underlying prices and other factors change—very important for risk and hedging) for your products in a Monte Carlo simulation. And even better, you will get it for all times in the simulation.

Differential programming is a generalization of deep learning, so all the same practical examples from deep learning apply. The difference, however, is that in DP you can write your "models" simply as functions. This means than in an ideal DP framework, the code you write needs no special types and no special syntax. This would be particularly useful if you're using some other library's code which was not written with DP/DL in mind at all (think ODE solvers, optimization routines, etc.); that code can be added to your "model" (again, just a regular function), and it will just work!

The difficulties of course lie in implementing such a framework, verifying whether your program is truly differentiable, and so on. The author of this blog post has written a working prototype framework in Julia called Zygote.jl. In short, it achieves DP through metaprogramming, applying source code transformation techniques at compile time, and it has already been quite successful.

The thought on top of my head is the giant set of default parameters that sit on top of pretty much every program, set by intuition by the original coders and then never touched again. Define a loss function, and you can start to optimize those parameters "for free".

Reply on news.ycombinator.com