> In contrast, symbolic differentiation instead would be to evaluate the entire program in terms of symbolic variables to produce a final expression for your quantity of interest as a function of the inputs, then differentiating that and substituting concrete values in.

This describes Theano exactly though... You build up a static, symbolic computation graph representing a mathematical function, and then you can call theano.tensor.grad(...) on that graph to get another computation graph, a symbolic representation of the gradient. Everything is completely symbolic at this point -- you could prettyprint the computation graph output by Theano's `grad` function and get exactly what symbolic differentiation seems to give you. You can then call theano.function(...) on either of those graphs to get an actual (Python) function which you can substitute concrete values into.

edit: in fact, Theano's own docs on its gradient-related functions [1] call it "symbolic differentiation" right at the top.

edit2: I decided to try this on the author's sin(x+...) example he uses as a pathological case for symbolic differentiation. Theano has exactly the behavior he warned that symbolic differentiation has -- you get a symbolic expression whose size grows with the loop range. It's quite large even with a size of 3 [2] (let alone 100).

[1] http://deeplearning.net/software/theano/library/gradient.htm...

[2]

    import theano
    from theano import tensor as T

    def f(x):
        y = x
        for i in range(3):
            y = (x + y).sin()
        return y

    x = T.scalar(name='x')
    my_function = f(x)
    print "my_function =", theano.pprint(my_function)
    my_gradient = T.grad(my_function, x)
    print "my_gradient =", theano.pprint(my_gradient)

->

my_function = sin((x + sin((x + sin((x + x))))))

my_gradient = ((((fill(sin((x + sin((x + sin((x + x)))))), TensorConstant{1.0}) * cos((x + sin((x + sin((x + x))))))) [... much much more]