Ensmallen – Flexible C++ library for efficient mathematical optimization | Hacker News Reader

Ensmallen – Flexible C++ library for efficient mathematical optimization

ensmallen.org

95 pointsforrrealman16 comments

I've written several frameworks for optimization and the approach here of defining special objective functions works quite well. I like the look of this and would love to try it out soon.

This is a small thing, but I'm really happy to see that their L-BFGS line search algorithm supports a maximum step size. Many numerical optimization libraries do not offer this simple feature, which is critical if your goal is to find a minimum that is close to the initial starting location. This is important in atomistic geometry optimization.

In terms of c++ matrix libraries, I've found eigen3 to be better than armadillo (although armadillo is easier to use).

Github is https://github.com/mlpack/ensmallen

There's also a paper: https://arxiv.org/abs/1810.09361

This is looking very nice, can't wait to try it out. But I feel it's a missed opportunity to not provide some simple examples [1], as well as a benchmark to quickly estimate the amount of flops that can be expected. The main reason to use a library like this one over pyTorch or autograd/scipy is speed, after all.

[1] The test cases can be seen as examples, obviously, but they are not written in an easily accessible form IMHO.

EDIT: Looking more closely, it seems you need to provide an explicit gradient, no autodiff included, which makes it a complete nonstarter for me.

Looked at the documentation, had to do a double take "Is this mlpack??? a fork??" then checked, yup, it's a header only version of the mlpack's optimization library. Very cool.

Anyone know how this would compare to IPOPT for constrained, sparse nonlinear optimization?

How does this differ from Ceres?

Ceres is great for sparse, structured non-linear-least-squares problems (that is, problems where levenberg-marquardt is applicable). I believe it was initially designed for bundle-adjustment problems arising from structure-from-motion applications.

Ceres also does work for dense problems and general convex optimization, but it doesn't provide as much of an advantage over other packages (that is, you could just call into any existing l-bfgs implementation and wouldn't benefit much).

For some other problems, Ceres doesn't scale or can get in the way. For example, if you have a single value to maximize and a large number of variables, then ceres' auto-diff system won't work (ceres::Jet is based on "forward-mode" automatic differentiation, storing partial derivatives on the stack. If you have a high-dimensional gradient, it can exhaust stack space). On the other hand, Ensmallen doesn't seem to come with any auto-diff functionality.

As far as I can see, Ceres accepts functions which calculate a residual vector, whereas ensmallen accepts only function which accept a scalar error. This is a subtle but important difference: The vector allows much faster convergence on a number of model fitting problems.

ensmallen also has support for batch optimization (like SGD), constrained problems, and SDPs for instance. The set of implemented optimizers is also quite large (40+ algorithms).

Sounds like the name is based on the perfectly cromulent word "embiggen"[1], but somewhat confusingly uses the first part of the word "enlarge".

[1] https://www.merriam-webster.com/dictionary/embiggen

It's not confusing, it's a generic pronunciation rule: 'n' before 'b' becomes 'm'.

>Search google for "cromulent" >Sees explanation from Wikipedia (https://en.wikipedia.org/wiki/Lisa_the_Iconoclast) >Look below, in "People also ask" section >"What does Embiggen mean?"

;)

I don't think "emsmallen" can be said reasonably :)

Huh, turns out you're right, which is entirely why em- and en- both exist and mean the same thing.

https://en.wiktionary.org/wiki/em-#Prefix

Reply on news.ycombinator.com