> If I had to guess, I’d say that dual numbers may be slightly slower than backpropagation
The cost of Dual numbers (a form of forward-mode differentiation) scales linearly with the number of derivatives (just like finite differencing, but more accurate). Backpropagation, or reverse-mode differentiation, is a constant factor times the cost of a function evaluation. For neural nets with millions of parameters, backpropagation is going to be millions of times faster than dual numbers.