Hacker News Reader

Can you get second order with only the gradient? Even fancy accelerated gradient descents are still first order. I suppose there are BFGS-like quasi Newton methods, but those are still building up estimates of the second order info over multiple iterations.

View on HN

You can get closer to second order if your error is a sum of squares of the form 0.5 * f^T * f, in which case you case compute the jacobian J of the fitness f, and approximate your hessian as J^T * J. That's the Gauss-Newton method and it's very effective if a linear approximation of f holds locally.

That's an approximation but that's arguably a second order method, since it acknowledges the nonlinearity coming from the squared norm of f.

Reply on news.ycombinator.com