Introduction to Automatic Differentiation

alexey.radul.name

25 points·noelwelsh··6 comments

Q: "What’s the derivative of a data structure anyway?" A: The Derivative of a Regular Type is its Type of One-Hole Contexts http://strictlypositive.org/diff.pdf

Amazing, I was looking for such a paper for a long time.

Automatic differentiation computes with so-called dual numbers, which store an infinitesimal (e) alongside the result of the computation. The infinitesimal is the derivative. Working with infinitesimals is how calculus started out but was abandoned when calculus was formalized using epsilon-delta limits (http://en.wikipedia.org/wiki/%28%CE%B5,_%CE%B4%29-definition...)

The is a newer formalization of calculus, called non-standard analysis, that works with infinitesimals, called the hyperreals. As you can imagine, there is a strong relationship between dual numbers and hyperreals. I found this exchange interesting:

http://math.stackexchange.com/questions/341535/is-the-theory...

It's kinda cool that non-standard analysis slips in through the back door when using dual numbers. I always found working (informally!) with infinitesimals quite simple.

So there's a new name for taylor expansions now?

It's a name that has been used for a long while in Numerical Analysis. After all, saying Taylor expansion doesn't help much when integrating DE's (in some sense even Runge-Kutta methods are a kind of Taylor expansion), for instance, whereas AD does give a lot more information.

AD is a way (actually ways) of organizing a computation. There are many different ways to organize the computation and evaluation of a Taylor series at a point.

As an exercise, try to explain the difference between symbolic differentiation, forward mode AD, and reverse mode AD. You will find that there is more content here than just a new name.

Reply on news.ycombinator.com