mfkasim1

It's here: https://github.com/machine-discovery/deer :)

Yes, sorry for that. I'm coming from physics, so L for sequence length and n for the number of dimensions make more sense :D

I agree with the cubic time and quadratic space is a big limitation for now and I'm looking for ways to make them linear (or close to linear).

We're preparing (i.e. cleaning up) the code for the repo. Will update you when we release the code.

Yes, there is no convergence guarantee, but what we found is that typical untrained RNN units (e.g., GRU, LSTM, or a simple MLP for NeuralODE) can converge within 3-5 iterations which gives them a huge speed up over the sequential method. The non-convergence typically happens after many thousand steps of training, but that can be addressed by saving the RNN output from the previous training step as the initial guess for the next training step.

Adding more context from the paper. Although there is no convergence guarantee in forward calculation, the gradient computation only requires 1 iteration and always converge (see section 3.1.1), so even though the forward calculation still uses sequential method, the acceleration in backward computation might be achieved with our method.

The author of the paper here. The cubic time and quadratic space complexity is with respect to the number of dimensions (n), not the sequential time steps. The time and space complexity w.r.t. the number of time steps (L) are both linear, specifically O(Ln^3) time and O(Ln^2) space. The method gives larger speed up with longer sequence (>1k time steps) but with relatively small number of dimensions (<= 64 dimensions from our experiments).