Hacker News Reader

Comment by Atrix256 | Hacker News Reader

BTW this older article of mine is extended with a new one that shows how to handle multiple variables (: http://blog.demofox.org/2017/02/20/multivariable-dual-number...

View on HN

> If I had to guess, I’d say that dual numbers may be slightly slower than backpropagation

The cost of Dual numbers (a form of forward-mode differentiation) scales linearly with the number of derivatives (just like finite differencing, but more accurate). Backpropagation, or reverse-mode differentiation, is a constant factor times the cost of a function evaluation. For neural nets with millions of parameters, backpropagation is going to be millions of times faster than dual numbers.

Thanks!

http://www.denizyuret.com/2015/02/beginning-deep-learning-wi...

This guy did neural nets with dual numbers in julia and found it to take ~1.2 times as long as standard backpropagation, which suggests that if you had dedicated hardware for it it would be very nice.

I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation.

Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.

Huh. I stand corrected. I thought for sure the package used automatic differentiation using dual numbers.

Reply on news.ycombinator.com