That is, I can see how the derivative can help optimise a simulation, but I'm not clear on how using an auto derivative really helps.
Edit: note that I also see a difference in being able to get the derivative of a function, versus writing in a sense where any function can be moved to its derivative.
Finding a derivative by hand gets tiring fast, and is error prone. Used to be a neural net paper doing anything novel spent like 1-2 pages deriving the derivative of its cool new thing. Not it's derivative isn't even mentioned.
An example of a weird use case outside ML is AD allows you to differentiate through a raytracer/complex program. Maybe your raytracer has a couple parameters for lighting and you have a target image you can to create something as similar to as possible. You could use AD to optimize the lighting parameters. That's one problem that mathematica will be very unlikely to be able to do. For a large application if you want to AD the entire thing either you are using a framework that supports AD everywhere like tensorflow/pytorch or you need language level support like Julia. Pretty few languages have AD at the language level.
To try and rephrase, the techniques used are similar in both. In that it is all calculus. However, with AD, the focus is reducing all calculations down to what was performed in expressions that have duals, and getting the derivative of that, as calculated, to use in making a choice.
Doing this symbolically would require the entire function space be solved for the problem in very difficult ways that are not easy to avoid.
That is, with AD, you don't get the total derivative of a problem, per se. Instead, you get the problem reduced to a a method that can evaluate at a tuple, where you also have the derivative for that tuple?
Extrapolating from this, even if a problem had piecewise spots where it will not differentiate, AD mostly works around this by focusing on the pieces?
Symbolic Differentiation, like Mathematica tends to start to slow down and generate massively amounts of code once functions get complicated. I am also not sure that it can handle dynamic length loops.
Source to Source AD, like Zygote (Julia), Tapenade (Fortran), Jax (Python), Enzyme (LLVM) have a lot of what you might want though.
Would love to see a blog or other post on solving this kind of problem. One with an inner LP would be amazing to see. I'm pretty sure some of the stuff I've looked at in the past on supply chain solutions could find relevance here.
I don't get how it helps with some discontinuity problems, but I think I can get how those aren't as important in many contexts.