Oh well, time to study this intensively. Here's a link for the curious: https://arxiv.org/abs/1411.6671
I've never really thought about it in terms of your comment, however, which is actually very motivating. Thanks for the perspective random internet person.
(Apologies for the OT comment)
It vindicates your thinking in that it indicates that you are picking up the techniques that professionals use to generate ideas. You just have to accept that as you're learning, most of the ideas will have been generated and addressed.
What gets awkward is when you're doing interdisciplinary research and you're using the generative mechanisms from one discipline in the other. By and large both sides stare at you in confusion, or it requires multiple hour conversations to lead the other discipline's practitioners into a mental space where they can say, "Oh, yeah, that wouldn't be useful because of X." It makes the process of tuning your idea generation much slower. On the flip side, you hit the unknown way faster, sometimes great vistas of unknown where everyone goes, "Umm...wow. Yeah. Never heard of that continent. Let us know what you find."
Advise to new PhD candidates: take a boring idea and write the best paper you can on it. It will be 10x better received than your "great" idea.
If you wanted to train a neural network model that would generalize, you would need build in physical constraints into the model architecture. For example, dynamics are governed by a Hamiltonian based on pairwise interactions, which should be integrated as an ODE. A nice recent example of this from folks at DeepMind is this paper “Hamiltonian Graph Networks with ODE Integrators”: https://arxiv.org/abs/1909.12790. That said, if you go down this road too far you may find that you are just learning an approximation to Newton’s law of gravity, which we already know exactly!
The interesting work in this space is finding appropriate interpolation problems inside the context of state of the art physics based models. In any complex simulation, there are tunable parameters and approximations that are suitable to replace with ML. If I were working on this problem, I would start by making a differentiable version of Brutus, and try to deeply understand its strengths and weaknesses.
Neural nets are just high dimensional interpolation. The only reason why they are more powerful than “classical” ML is that you can fit them when embedded in an arbitrary computational graph, in which it is easy to embed prior knowledge (“differentiable programming” style). If you’re not doing that, you might as well just stick with the blackbox algorithms of Scikit-Learn.
In terms of practical applications I don't see what's the point of training a network on simulations produced by deterministic systems. But I guess they didn't really mean their technique as a replacement for deterministic systems because they say they envision it as a sort of crutch were Brutus falters.
They don't evolve the simulation very long and the errors rapidly increase with time. See Figure 3 for typical errors of about 20% on the validation set with clear signs of overfitting. https://arxiv.org/pdf/1910.07291.pdf