Hacker News Reader

Comment by efavdb | Hacker News Reader

Yes sometimes it’s hard to measure a derivative. Eg when doing hyperparameter tuning in ML, you can read out a metric at a given choice of parameters, but it’s generally not easy to get a gradient.

Shameless plug: I happen to have recently written a package for the opposite limit! It finds roots when you can only measure the derivative.

https://github.com/EFavDB/inchwormrf

View on HN

You might be interested in David MacKay's old conjugate gradient implementation in C that also only uses derivatives: https://www.inference.org.uk/mackay/c/macopt.html

(It was written in another time, don't expect it to be nice. My 20-year-old Octave/Matlab wrappers on that page have almost certainly bit-rotted, don't expect them to work.)

Thank you! I’ll definitely check this out.

I assume backprop is still most efficient for the neural net itself, rather than one of these algorithms? I am doing a course and I am aware that there are other algorithms than gradient descent but haven’t seen the details yet on that.

I know things like cross entropy loss are designed to be easy to differentiate and probably more efficient to do that than something that approximates it.

Yeah, backprop gives efficient gradient calcs to train at a given net architecture. To find the best architecture though people try various other methods, eg random search or Gaussian processes, which don’t evaluate derivatives wrt architecture Params.

> It finds roots when you can only measure the derivative.

How is that even possible? The function f(x)=x-7 has a single root at x=7. Its derivative is f'(x)=1. How do you recover 7 from 1 ?

You need one initial value to start from

Reply on news.ycombinator.com