Hacker News Reader

Comment by ChrisRackauckas | Hacker News Reader

As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities.

View on HN

Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference. What's interesting is that not only do preliminary results yield significantly less error in prediction vs RNNs. Differentiable models allow for extrapolations well beyond range of observable test data set.

Neural Ordinary Differential Equations

https://arxiv.org/pdf/1806.07366.pdf

I am working on very different approaches with very different applications, but yes the neural ODE (not a solver BTW) is a good example of this kind of application and the effectiveness of the approach. It's a fantastic paper if anyone hasn't read it.

Can you define what is pervasive automatic differentiation please.

AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).

Something like eager execution in tensorflow?

Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.

Eager execution in TensorFlow doesn't work on all Python code or all Swift code. What I am essentially asking for is equivalent to sticking AD on the source code of say any part of SciPy and expecting it to work without hard coding workarounds.

Here's my deal. I wrote and maintain the differential equation solver library in Julia. This thing is a huge piece of code composed of almost 70 packages and took something like 5,000 commits and almost entirely pure-Julia. I will keep writing and maintaining this code as its own research project for many reasons. And, sometimes, I want to AD this code or stick it in a neural network.

It's a very non-standard application of this kind of code, so it wasn't built to do this from the start. But I am a greedy hacker. I don't want to have to re-write it onto some computational graph package (TensorFlow) or build interfaces. I want to just call some AD library function on any pure-Julia function in my package and have it output derivatives just like it was any numerical diff package. ForwardDiff.jl, ReverseDiff.jl, Flux.jl, etc. all have autodiffs that work on these routines. I find that magical. In fact, I was shocked that it can be faster than the standard way of calculating these derivatives via sensitivity analysis (we'll put a paper out on Arxiv in a few days about this). There's still a few hiccups that can be fixed up for non-ML applications, but these Julia differentiation tools have really impressed me.

> faster than the standard way of calculating these derivatives via sensitivity analysis

This is quite shocking indeed, I'll look for the paper when it comes out. I'm particularly interested in whether this is true for the sensitivity functions used in shape optimization of composite materials. For example, when optimizing for a stiff, conductive material, eg https://www.sciencedirect.com/science/article/pii/S002076830...

This is it: https://arxiv.org/abs/1812.01892

Thanks! At a quick glance looks like this may not be useful for the problem I had in mind, but I'll have to read it in more detail.

> As someone who works in merging differential equations and machine learning

That sounds really interesting, could you expand on your work/research?

A bit of googling yields http://chrisrackauckas.com/ which seems like a likely match.

Reply on news.ycombinator.com