duvenaud

I think everyone who uses probabilities to make decisions agrees that all our models are flawed in many ways. But how else do you propose to weigh different possibilities?

As for your proposal for how to spend money, I agree those are fruitful and promising directions. I think the fields of public choice theory and mechanism design are working on some of the problems you mention.

> But with a given Jacobian function there's no finite differences or forward mode.

Right, but instantiating an entire Jacobian is always going to scale at least quadratically with time. The point I was trying to make is that the existing non-adjoint approaches were never going to scale to large systems with millions of parameters. This is the main attraction of reverse-mode, and it appeared to me that this was a major obstacle for fitting large models using existing packages (excepting CasADi).

> I'm surprised you'd attempt to solve the equation backwards like that because it's known to not be stable.

> it's a well-known result that the method drifts from the true solution doing a backwards integration, so the values so z(t) needs to computed with forward passes in order to be correct.

I agree that a purely reverse-mode gradient solve will diverge from the forward trajectory to some degree. But to say the gradients are 'correct' or not seems a bit strange to me. Every numerical solve introduces some degree of error, and I think the most useful discussion to have is the tradeoff between computation cost and numerical error. Re-solving the system forwards is one strategy to reduce error at the cost of computation. Another strategy would be reducing the error tolerance of the reverse solve. There are situations where our strategy might give worse precision wrt the parameter gradients for a given computational budget, but I wouldn't dismiss it out of hand. Especially since it's about as computationally cheap as one could hope for - O(1) memory and similar time cost as the forward solve. Also, it worked for our applications.

> Any statement on it is too broad to be useful.

I appreciate the detailed reply. But is there anything you can say about when to try implicit methods over explicit? What was the motivation for developing implicit methods in the first place?

Thanks for the detailed remarks, and for pointing out that my statement above about the novelty of our implementation was wrong - CasADi uses the same algorithm as was released in 2013. My apologies. Joel Andersson pointed this out to us and we added a cite to his thesis several months ago.

As for Sundials and FatODE, my understanding was that they used finite differences, forward mode, or differentiating the solver operations for at least some aspect of their sensitivity analysis.

On another topic, I think you might be misunderstanding how we're running our adjoint sensitivity analysis. You say

> In the Neural ODE paper, to do a reverse solve of the adjoint ODE it solve the forward ODE from the beginning time point until the point. Clearly, this is really slow because it requires a lot of forward solves over long intervals.

This isn't true - when we do the reverse solve, we get all gradients using a _single_ solve going backwards in time. I'm now realizing that the misunderstanding might be caused by our Fig. 2, which shows that multiple backward solves are necessary when the loss depends on the state at multiple time points.

I agree that our statement about the stability of implicit over explicit was overly broad, thanks for pointing that out. Can you suggest a more accurate statement about the advantages of implicit over explicit methods?

I also agree there is a lot of numerical work to be done in this area, and I'm glad that people more knowledgable than us (such as yourself) are looking at it too!

Well, you can differentiate through the operations of ODE solvers written in any framework that has autodiff. But the whole idea of adjoint sensitivity analysis is to not differentiate through the operations of the solver, to save memory and to control numerical error more directly.

Whoops, I just realized that I failed to qualify that statement properly. CasADi [https://web.casadi.org/], released in 2013, uses adjoint sensitivities to compute gradients through ODE solutions using symbolic (but still reverse-mode) differentiation.

I should have said something like: We were the first to implement adjoint sensitivities in a modern, tracing-based autodiff framework suitable for machine learning [HIPS autograd]. But I messed up and should have given credit to Joel Andersson for being first, my apologies.

It does, but that's just a coincidence. That's a visualization of the latent space for the toy time-series problem in our paper that has to learn to model spirals of different sizes and orientations.

Ah, that would stop certain dimensions from changing, but the output would still be the same size.

Section 7 is titled "Related Work".

1. Great question! You're correct that standard NF isn't efficiently invertible. CNF is, and we wanted a fair comparison. So for this experiment, we reversed the direction that NF transforms the data, so that it goes from the data to the latent space. Training this way means that you can't use the resulting model as a generator, but it at least let us compare likelihoods with CNF for this paper.

2. We had to set the error tolerance relatively small during training to keep the gradients stable. I don't think we used any fancy initialization tricks, but to be honest I have to ask Ricky Chen and Will Grathwohl, who ran all the FFJORD experiments.

My understanding is that the advantage of something like a SNN is mainly in custom-hardware ultra-lower-power applications. But it adds a lot of complexity and makes training a lot more difficult, compared to something like ODE nets or standard discrete neural nets.

Looking at the links you provided, they don't appear to train the network dynamics, only the feedout weights. In principle you could differentiate through the ODE solvers used in the software package linked to train the dynamics, but as far as I know we were the first people to release an open-source set of differentiable solvers using reverse-mode adjoint sensitivities (the most scalable variant of autodiff for training scalar losses). [https://github.com/rtqichen/torchdiffeq]

In the paper we talk a bit about how you can now more easily train Possion process likelihoods, which might make training SNNs easier. I'm not sure.

None in this paper, which is just showing proofs of concept. But our follow-up paper does have SOTA density modeling among the class of efficiently-sample-able generative models. This is because the design of continuous flow models is less constrained than for discrete flows. But other than that, it's still early days for this whole model class.

You train these models in the same way you train a neural network, with stochastic gradient descent. For supervised learning, its main benefit is extra flexibility in the speed/precision tradeoff. For time-series problems, we expect this will ultimately allow us to handle data that's collected at irregular intervals - but we haven't yet moved past the prototype stage in that setting.

Interesting question. There might be, but that's more a question of fitting parameters. This paper was about a different way to set up a parametric model, that can be trained in the usual way. I don't think the fact that internally it uses ODEs changes its susceptibility to catastrophic forgetting in the online learning setting.

1. As the other reply said, we just mean an ODE solver whose internal operations are a detail we don't need to worry about.

2. This paper only had proofs of concepts and toy demos. But our follow-up paper, FFJORD [https://arxiv.org/abs/1810.01367] we used the continuous normalizing flows to get SOTA efficiently sample-able density models.

I did a CS undergrad at the University of Manitoba. Then took some time off to do a startup and was in the army reserves. Then when to UBC to do an MSc in CS + Stats. My Phd was officially in the made-up-sounding subject of "Information Engineering" at Cambridge, but really I just worked on Bayesian nonparametrics the whole time. I didn't start working on deep learning until my postdoc.

The comparison we make is to residual networks, which I think is valid. First, we do parameterize a theta that changes with time, using a hypernet. But this is equivalent to the way sampo wrote the equations above - you can just feed time as another input to the dynamics network to get dynamics that change with time.

Second, I agree that general feedforward nets allow dimension changes, but resnets don't. This model is a drop-in replacement for resnets, but not for any feedforward net. If we gave the wrong impression somewhere, please let us know.

We didn't make the analogy with RNNs, because I don't think it fits - standard input-output RNNs have to take in part of the data with every time step, while here the data only appears at the input (depth 0) layer of the network.

Good question. The model developed in that paper, (neural autoregressive flows) is a discrete-layered architecture. It's a member of the Normalizing Flows family of models. Normalizing flows define a parametric density by transforming a sample from a Gaussian by a series of transformations:

  z0 ~ Normal(0, I)
  z1 = f1(z0)
  z2 = f2(z1)
  z3 = f3(z2)
  x  = f4(z3)

and they use the change of variables formula to compute p(x):

  log p(x) = log p(z0) + log det | df/dz0 |

In our paper, we propose a continuous-time version of normalizing flows, called Continuous Normalizing Flows. We derived a continuous-time version of the change of variables formula:

  dlogp(z(t)) = -trace(df/dz)

Anyways, the confusion is probably that both models are called flows. We wish that Normalizing Flows had used a different name, so that we could save the word "flow" for continuous-time transformations, but it's too late for that :)

Neural autoregressive flows are powerful density models, but are computationally costly to sample from. Continuous normalizing flows cost about the same to evaluate densities and to sample. We compared the two in a follow-up paper: [https://arxiv.org/abs/1810.01367]

Ooo, great question. In both cases, we define a function using a sum over an infinite number of infinitesimal things.

In the GP case, those things are random and independent of each other, so the central limit theorem applies and we get a simple Gaussian.

In the ODE case, the infinitesimals are deterministic and depend on the previous ones in the sequence, so the final answer is deterministic and impossible to compute exactly in general.

You could also use a GP to model the dynamics of an ODE, and this was done recently: [https://arxiv.org/abs/1803.04303] although the drawback was that they couldn't train the model by backpropagation.

The closest thing to a continuous-time RNN that exists are Neural Hawkes Processes [https://arxiv.org/abs/1612.09328]. They use a different model where observing the system necessarily changes its state, which is natural in some settings but doesn't fit in others. On the other hand, their model scales today and ours doesn't :)

Sure thing. A few years ago, everyone switched their deep nets to "residual nets". Instead of building deep models like this:

  h1 = f1(x)
  h2 = f2(h1)
  h3 = f3(h2)
  h4 = f3(h3)
  y  = f5(h4)

They now build them like this:

  h1 = f1(x)  + x
  h2 = f2(h1) + h1
  h3 = f3(h2) + h2
  h4 = f4(h3) + h3
  y  = f5(h4) + h4

Where f1, f2, etc are neural net layers. The idea is that it's easier to model a small change to an almost-correct answer than to output the whole improved answer at once.

In the last couple of years a few different groups noticed that this looks like a primitive ODE solver (Euler's method) that solves the trajectory of a system by just taking small steps in the direction of the system dynamics and adding them up. They used this connection to propose things like better training methods.

We just took this idea to its logical extreme: What if we _define_ a deep net as a continuously evolving system? So instead of updating the hidden units layer by layer, we define their derivative with respect to depth instead. We call this an ODE net.

Now, we can use off-the-shelf adaptive ODE solvers to compute the final state of these dynamics, and call that the output of the neural network. This has drawbacks (it's slower to train) but lots of advantages too: We can loosen the numerical tolerance of the solver to make our nets faster at test time. We can also handle continuous-time models a lot more naturally. It turns out that there is also a simpler version of the change of variables formula (for density modeling) when you move to continuous time.

Thanks! If I understand your question, you're asking if the ODE time-series model we proposed works best in things like idealized frictionless or gravity-less setting? The answer is that we expect it to work (in principle) even for messy, non-physics-experiment-style situations. This because we don't model the system dynamics directly, but in a latent space that we learn. So even in a messy situation like health monitoring of humans, as long as the data are being driven mostly by hidden factors that are changing smoothly through time (like overall health, infections, hormones or whatever) the neural ODE can abstract away from the messy data. This is all still just in principle - there are a few technical challenges to scaling these models up. But nothing that looks insurmountable right now.

Senior author here, I'm happy to answer any questions.

We just released source code: https://github.com/rtqichen/torchdiffeq . This includes PyTorch implementations of adaptive ODE solvers that can be differentiated through automatically. So you can mix and match these ODE solvers with any other differentiable model component.

There's already been a bit of follow-up work, turning Continuous Normalizing Flows into a practical generative density model: https://arxiv.org/abs/1810.01367

And now we're mainly working on 1) Regularizing ODE nets to be faster to solve and 2) getting the time-series model to scale up and extend it to stochastic differential equations.

That diagram is just meant to show what type the inputs and outputs of the neural network are. I'm not a chemist myself, but the other two authors are.

Yes, the datasets were the real limiting factor for this project, and as you point out, reactions depend on many more things than their reagents. Hopefully this work will inspire someone to build a better dataset of reaction setups and outcomes!

Thank you! I guess all you need to do to get chemistry on hacker news is add neural nets to it.

I honestly don't know much about synthesis or the state of computationally assisted inverse design of molecules, except that it seems like a great idea, and that it is still early days. As you say, proper formulation of the guidelines is still necessary - as far as I understand, right now the synthesis people still use a lot of judgement at each step, and all these little choices will have to be recorded to build a useful dataset.

Thanks for the kind words! I would agree that the function we're trying to learn is very complicated, with many exceptions. But that only improves our comparative advantage, at least compared to novice chemists. We might also be able to help our system take advantage of the knowledge of physics that expert chemists use to predict reaction outcomes by giving our network access to the output of a physical reaction simulator.

Extending the system to (try to) predict yield percentages or chirality is straightforward. The hard part, in my mind, is that there aren't a fixed number of reaction types. As molecules get bigger, we'll have to move away from predicting one of a fixed set of reaction types, to directly predicting products - but this is a much harder problem.

Our system probably isn't good enough to design synthesis pathways yet, but that is the eventual goal. A system that also predicted yield would of course help with that, and that would be another straightforward extension.

Good points. However, even if chemists already know how to predict reactions, giving this knowledge to a machine will allow a much faster search over possible synthetic routes.

I agree that reaction outcomes depend on many other factors besides the reagents. In the future, I'm sure we'll create reaction prediction frameworks that also take these other factors as inputs. The problem right now is that there aren't many datasets that include these extra factors.

We're not advocating turning retrosynthetic analysis over to machines yet. These are just baby steps.

One of the authors here. Ask us anything!

This paper is just a first step - what we'd really like to use this for is designing recipes for synthesizing new molecules.

I would also be remiss if I didn't link to a closely-related paper from another group that came out at the same time: http://pubs.acs.org/doi/abs/10.1021/ci5006614

Thank you very much, Max! To answer your questions:

- As you say, jointly training the autoencoder for reconstruction and prediction accuracy would be easy to code, but it might be tricky to get the tradeoff right since we usually have relatively little labeled data. A massively multi-task approach probably makes sense here. I'm not sure how hard it will be to get smoothness w.r.t. the latent representation, but the logP result was mildly encouraging.

- I agree that it's probably time to bring some of the recent augmented recurrent network architectures to bear on graph decoders. What makes is hard is that it requires differentiating through a cascading series of discrete choices, but people are making progress on this sort of problem.

- I do think we need to completely start over on the synthetic accessibility. There are just too many ways for molecules to be weird to try to write an explicit function for it. In fact I'd go so far as to say that this is the 'missing half' of this method.

- I view this method as a complement to physics-based simulation. A neural net is almost always going to be cheaper to evaluate than a physical simulation, so it's not a bad way to compile all the simulation results together.

One of the authors of the original paper https://arxiv.org/abs/1610.02415 here.

From a machine learning point of view, we simply glued together two techniques: text autoencoders and Bayesian optimization. That is, we trained an autoencoder to transform a text representation of chemicals (SMILES) to and from continuous vectors. Then “chemical design” is just maximizing a function of a continuous variable, something that we already know a lot about.

We also showed off some of the nice things that one can do with continuous latent representations, such as interpolation. This had already been done for images by many people, and for text in https://arxiv.org/abs/1511.06349

Of course, our paper is just a proof of concept. For instance, instead of encoding to and from SMILES, it would be much better to encode to and from graphs directly. We know how to encode graphs into vectors, but I don’t know of a good way to decode a vector into a graph.

Another open problem is that it’s hard to know what to optimize. Our initial experiments optimizing for specific chemical properties produced suggested molecules with crazy structures, such as giant rings. Human chemists have a great intuition for what is easily synthesizable or stable, and it’s hard to articulate all the properties we want the molecule to have programmatically. Alternatively, we could enforce the optimizer to only look at molecules similar to ones we’ve already seen, but this is unsatisfying too - after all, the best result of exploration is when you find something unlike what you’ve seen before.

As far as I understand, the adjoint method computes gradients for functions that obey hard constraints (such as fluid solvers), and the main advantage is that it avoids differentiating through iterative constraint satisfaction procedures.

I had some success with naively differentiating through fluid solvers, though. Here is a fluid field whose initial velocities have been initialized so that it will end up matching a given image (i.e. blowing a fancy smoke ring:)

https://github.com/HIPS/autograd/blob/master/examples/fluids...

and here's a free-form wing shape in the middle of being optimized to maximize lift-to-drag ratio:

https://github.com/HIPS/autograd/blob/master/examples/fluids...

I too find it baffling when I see engineers doing gradient-free optimization of simulated objective functions, although it's not always easy to compute gradients, especially when using massive Fortran codebases or very large-scale simulations.