My first forays into making neural networks used replacement rules to modify an expression tree until all the “D” operators went away, but that takes exponential complexity in network depth if you aren’t careful. Finite differences is linear in number of parameters, as is differentiation by Dual Numbers

Backprop is the application of dynamic programming to the chain rule for total derivatives, which sounds trivial only in retrospect.

You can do forward propagation. Humans typically finds forward easier than backwards.

since you asked ... how about Monte Carlo with Gibbs sampling?