Hacker News Reader

Comment by tachim | Hacker News Reader

How does SigOpt compare to GPs?

Great question. SigOpt is a hosted, scalable ensemble of different Bayesian optimization methods. Many of these methods use Gaussian Processes (GPs) as part of the modeling aspect of their Sequential Model Based Optimization (SMBO). We've written up a primer on Bayesian optimization [1] that goes over some of the different methods, with lots of citations for diving deeper.

We've found that SigOpt compares very favorably [2] to other Bayesian optimization approaches. In addition to this, our hosted platform allows people to harness the full power of GP backed Bayesian optimization with just a few lines of code [3] instead of the sometimes heavy administration required by other methods.

[1]: https://sigopt.ninja/1470077644/pdf/SigOpt_Bayesian_Optimiza...

[2]: http://arxiv.org/abs/1603.09441

[3]: https://sigopt.com/docs

Thanks. Do you guys do any meta-learning based on information from optimization runs on the many instances that SigOpt runs on? E.g. if Alice uses Sigopt for a neural net architecture and Bob does too, is the information from Alice's run used to improve Bob's?

We don't explicitly share information at the experiment level like that for privacy reasons. All user data is explicitly isolated from other users, on top of the fact that due to the black box nature of our optimizer we often do not know what the underlying method or system being optimized actually is.

We do, however, run a rigorous evaluation framework [1] over our methods as we iteratively improve (and compare to other techniques). This allows us to build up our ensemble of optimization strategies to most efficiently tackle problems that are most important to our users. As we see users leveraging our service for certain types of problems (like mixed continuous/categorical + failure regions) we do try to incorporate them more into our testing, roadmap, and ensemble, but only at the meta level.

[1]: http://arxiv.org/abs/1605.06170

Reply on news.ycombinator.com