Hacker News Reader

Comment by sega_sai | Hacker News Reader

Interesting, but it does not seem to be an overview of gradient optimisers, but rather gradient optimisers in ML, as I see no mentions of BFGS and the likes.

View on HN

I'm also curious about gradient-less algorithms

For non deep learning applications, Nelder-Mead saved my butt a fees times

It's with the utmost humility that I confess to falling back on "just use Nelder-Mead" in *scipy.optimize* when something is ill behaved. I consider it to be a sign that I'm doing something wrong, but I certainly respect its use.

Nelder–Mead has often not worked well for me in moderate to high dimensions. I'd recommend trying Powell's method if you want to quickly converge to a local optimum. If you're using scipy's wrappers, it's easy to swap between the two:

https://docs.scipy.org/doc/scipy/reference/optimize.html#loc...

For nastier optimization problems there are lots of other options, including evolutionary algorithms and Bayesian optimization:

https://facebookresearch.github.io/nevergrad/

https://github.com/facebook/Ax

SAMBO does a good job of finding the global optimum in a black-box manner even compared to Nelder-Mead, according to its own benchmark ...

https://sambo-optimization.github.io

ChatGPT also advised me to use NM a couple of times, which was neat.

Look into zeroth-order optimizers and CMA-ES.

I think the big difference is dimensionality. If the dimensionality is low, then taking account of the 2nd derivatives becomes practical and worthwhile.

What is it that makes higher order derivatives less useful at high dimensionality? Is it related to the Curse of Dimensionality, or maybe something like exploding gradients at higher orders?

In n dimensions, the first derivative is an n-element vector. The second derivative is an n x n (symmetric) matrix. As n grows, the computation required to estimate the matrix increases (as at least n^2) and computation needed to use it increases (possibly faster).

In practice, clever optimisation algorithms that use the 2nd derivative won't actually form this matrix.

Reply on news.ycombinator.com