throwaway39210

Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).

> In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.

This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into.

edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top.

edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100).

[1] http://deeplearning.net/software/theano/library/gradient.htm...

[2]

    import theano
    from theano import tensor as T

    def f(x):
        y = x
        for i in range(3):
            y = (x + y).sin()
        return y

    x = T.scalar(name='x')
    my_function = f(x)
    print "my_function =", theano.pprint(my_function)
    my_gradient = T.grad(my_function, x)
    print "my_gradient =", theano.pprint(my_gradient)

->

my_function = sin((x + sin((x + sin((x + x))))))

my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]

> Put another way, when people say they're using symbolic or automatic differentiation to "calculate" the derivative of a function, "calculate" means different things: in the one, they're "calculating" the exact, symbolic expression for the derivative (which can optionally be evaluated); in the other, they're "calculating" the floating-point approximation of the values of the derivative.

If this is correct, then both Theano and TensorFlow are definitely doing symbolic differentiation, no? The `grad` function in Theano (and the equivalent in TF, not sure what it's called) takes a computation graph (a symbolic expression of a mathematical function) as input and produces a computational graph expressing the gradient (which as you said, can then optionally be evaluated at specified points). So maybe the author wasn't referring to these types of frameworks when he said there have recently been an explosion of these autodiff tools? Or are Theano and TF and the like, in fact, autodiff tools as opposed to "symdiff" tools, but I'm still missing the point somehow?

Can someone further clarify the distinction between automatic and symbolic differentiation?

The author claims that using symbolic differentiation on his example (f(x)=sin(x+sin(x+...+sin(x+sin(x))...))) builds a "huge expression that would take much more time to compute" than automatic differentiation. At the top of the article there's also an update noting an "explosion of great tools for automatic differentiation" recently, which I assume is referring to machine/deep learning frameworks like Theano, TensorFlow, etc. However, from my understanding, building an entire computational graph to express the gradient via the chain rule (backpropagation) is exactly what these frameworks do, and by the author's brief description that seems to fall under symbolic differentiation. So which one are these frameworks doing -- symbolic or automatic differentiation? And, concretely, how would they differ if they were instead doing the other one?

Edit: I've been doing some further reading trying to understand this, and it's only confusing me further. Wikipedia has this diagram [1] which supposedly illustrates the relationship between the two -- what the hell is going on here? The only interpretation I can come up with from it is that automatic differentiation takes code (written in a computer programming language) as input and produces code to compute the gradient, whereas in symbolic differentiation, the input and output are mathematical expressions. But this doesn't seem like a meaningful distinction to me...both math and computer code are languages; functions written in programming languages are mathematical functions (the ones that can actually be correctly called "functions", anyway). Someone please help me out here; what am I missing?

[1] https://en.wikipedia.org/wiki/Automatic_differentiation#/med...