What Is Differentiable Programming?

The basic idea is to have an expressive programming language where all constructs are differentiable. Since the composition of diffeomorphisms is a diffeomorphism, large programs (like ray-tracers) will be differentiable as a result.

Supervised learning is a classic inverse problem -- you know the desired outcome, but not how to get there. (think control systems)

Suppose you had a flexible model in whose parameter space you could search. A naive algorithmic approach would be to treat this as a brute force search problem. Maybe you can be slightly more clever algorithmically (better search algorithms, heuristics, etc)

But can you somehow be really clever and exploit the fact that the parameter space is continuous, with a smooth cost function? That's what gradient descent basically is. Lots of bells and whistles on top, to exploit extra problem structure.

Now, let's go back to our family of models. They just need to be transparent to gradient information -- then you could use simple ideas at the heart of adaptive control theory. Suddenly, the fact that autodifferentiation techniques can differentiate arbitrary computational algorithms (sibling comment explains how compositionality helps this) sounds mouth-watering! That's what makes a lot of people excited -- your model family just expanded to programs performing arbitrary computations on continuous parameters (including loops!) -- not just some silly function approximator like a natural network (exaggerating to make my point)

Let me try to give a high-level explanation. As we have had success with using neural networks for different tasks, people began to ask what are the properties that make neural networks work so well.

When I mean work well, I mean a family of models which are generally easy to describe as well as be generally easy to train. Nearly all neural network models fit this bill as a they can be described with a loss function. A loss function being something which scores how accurately the current NN performs with a particular set of parameters. This loss function is then usually differentiable, and thanks to autodifferentiation tools, you can obtain the derivative of this function with respect to some parameters in this function. With this derivative, you can go ahead and run SGD to find the optimal parameters. We have now both empirical and theoretical evidence that SGD works fairly well for optimizing functions, particularly the high-dimensional ones neural networks describe.

So, the insight is we need a name to describe all these models which are differentiable and nice to train because of that. So now we can say what really ties CNNs, MLPs, RNNs, etc is not some biological metaphor, but that these are all expressible as some loss function we can find the minimum of using SGD.

> Could someone (...) provide a different explanation of the concept?

Yes. It has nothing to do with deep learning, and nothing to do with optimization (although it is useful for both). It is just a way to write your functions so that they are easy to differentiate.

in that regard it’s mostly in the first few paragraphs but it’s subtley stated...

If you have a model and somecparameter space on it your trying to optimise then you can’t do much other than search the whole parameter space or approximate such a search intelligently.

However, if your model is differeniable with respect to its parameters then it means you’ve got gradient ... and if you’ve got gradient then you can cut out a lot of searching because you can use gradient ascent and other such techniques .

So basically: differentiable means can be differentiated which means has gradient which means search less.

The rest of the article basically says that’d be nice because there is a way to think about differentiability modularly and therefore there may be a route to Lego things together from across fields to make uber models which still optimise well but use specialist knowledge.

The parameter space stuff just means that less knowledge means more data and more parameters which is bad.

Differential programming is a generalization of deep learning, so all the same practical examples from deep learning apply. The difference, however, is that in DP you can write your "models" simply as functions. This means than in an ideal DP framework, the code you write needs no special types and no special syntax. This would be particularly useful if you're using some other library's code which was not written with DP/DL in mind at all (think ODE solvers, optimization routines, etc.); that code can be added to your "model" (again, just a regular function), and it will just work!

The difficulties of course lie in implementing such a framework, verifying whether your program is truly differentiable, and so on. The author of this blog post has written a working prototype framework in Julia called Zygote.jl. In short, it achieves DP through metaprogramming, applying source code transformation techniques at compile time, and it has already been quite successful.

The thought on top of my head is the giant set of default parameters that sit on top of pretty much every program, set by intuition by the original coders and then never touched again. Define a loss function, and you can start to optimize those parameters "for free".

There are examples quite far from neural networks, the ones I can think of are broadly optimisation problems:

Many physics problems involve trying to find a function which minimises something -- energy, entropy, action. Or the state which makes the difference between two things zero. Sometimes adjusting many parameters slowly down the gradient is a good way to find these.

In bayesian statistics, the basic problem is to sample from a distribution, which you know only indirectly, by some kind of monte-carlo method. But the space to sample can be enormous. If I understand right, advanced ways of doing this exploit the gradients (of functions defining the distribution) to try to choose samples efficiently.

People have hacked tensorflow to do all sorts of things which its creators didn't intend. Or written tools specialised for another particular domain (like Stan). I guess the excitement is that instead of re-inventing the wheel in each domain, maybe this can be pushed down to become a language feature which everyone above uses.

For instance in robotics, the inverse kinematics could be formulated using the well-known equations, and combined with a CNN for vision, maybe do reinforcement learning across the whole thing. Or in audio classification, could use standard filterbanks construction for creating a time-frequency (spectrogram) representation, then combined with a CNN for classification.

Right now these things are done either: 1) Unstructured end-2-end learning, where a deep neural network has to discover the laws of physics unaided by structure. The models are massively over-parametrized, require a lot of data to train and vulnerable to adverse inputs. 2) As independent systems, optimized separately. If the first model is complex, the model that follows must usually reflect this complexity. A simpler global solution might exist, but cannot be found.

It is however super early days. Right now there are hints that going in this direction might be fruitful, but as far as I know, not many concrete wins or a lot of practice. That will take some years still.

In quantitative finance automatic differentiation will give you sensitivities (which tell you how the value changes when interest rates and underlying prices and other factors change—very important for risk and hedging) for your products in a Monte Carlo simulation. And even better, you will get it for all times in the simulation.

An analytic form of the derivative might not be able to be found by computer algebra or expressible in a compact form.

You can still do that, most of the times. But if your function has many conditionals the final expression may get cumbersome.

Sometimes, the symbolic derivatives will be faster to compute, but most often they will be slower than using dual numbers for example. It is just a different tool to solve the problem. Most often, it is easier to use: you do not need a computer algebra system, you just change the type of your numbers and call it a day.

> Why not use a Computer Algebra System instead for generating derivatives of given functions?

You will generate expressions, and evaluating those expressions leads to the execution of exactly the same elementary operations as with forward mode AD. But with the overhead of needing the Computer Algebra System, and storing the expressions. (Ok, in some cases the CAS is maybe able to simplify the expressions a bit.)

That's essentially what a source-to-source AD does, just with support for the extra features that show up in programming languages. For example, handling variable bindings gets you the typical Wengert list, and handling function calls gets you the Pearlmutter and Siskind style backpropagator (I wrote a bit about the relationships at [0]).

The short answer is that CAS systems work with a "programming language" that doesn't have these features and is therefore a bit too limited for the kinds of models we're interested in.

[0] https://github.com/MikeInnes/diff-zoo

CASes generate symbolic derivative expressions. This is typically a fairly expensive operation, and can be slow if the number of expressions are large or if the expressions are complicated. You have to pre-generate these expressions before you can do any work (paying an upfront cost). CAS'es also use a very general symbolic engine, which entails a performance penalty.

If you're doing numerical work and don't need explicit derivative expressions, automatic differentiation offers a method that is both fast and convenient.

When possible, one should link to the abstract page, if for no other reason than that it makes version tracking easier: Want, Wu, Essertel, Decker, and Rompf - Demystifying differentiable programming: shift/reset the penultimate backpropagator (https://arxiv.org/abs/1803.10228), a title which is entirely too cute for its own good.

They have some interesting software: https://feiwang3311.github.io/Lantern/

The only serious difference between classical AD and newer efforts like this is that they have an intermediate language ("what goes on the tape") with recursion and control flow operators which ideally lets them implement front-ends for normal programming languages.

You can achieve the same effect with classical AD by doing what PyTorch does in its default mode where the dynamic recursion and control flow is normal Python code and large tensor ops are asynchronously scheduled on the target device and an AD tape for that particular dynamic run is constructed each time. But the viability of PyTorch's approach is domain dependent: there are relatively few but large tensor ops, and while host-device synchronization is possible (for making the dynamic control flow dependent on intermediate device-computed results) it's also very costly. Although the first of these would be slightly less critical if the host language executed faster.

But, if you're not trying to achieve the kind of decoupling between host and device which PyTorch and similar frameworks want, then I don't see any particular reason you can't just use normal forward-mode or reverse-mode AD as an alternate scalar execution mode for existing code, which just needs a different interpreter/library, not a different intermediate language. For the purposes of differentiation, the branching and control flow operators don't exist, anyway. E.g. if you represent branching explicitly in your graph/tape as TensorFlow does, the subgradient of if(a, b, c) is just the subgradient of whichever of b or c is selected. The conditional variable a only serves as a selector; if you compute the subgradient with respect to the selector, the answer is always 0 (except for the knife-edge case where it's technically not defined but is treated as 0).

I think it's just the realization that the execution graph in machine learning models at this point are not really different from any programming language AST, which means there is potential in exploring the intersection between writing programs and writing machine learning models with the AD tools.