Unsupervised Learning with Even Less Supervision Using Bayesian Optimization
blog.sigopt.com
blog.sigopt.com
We're using Bayesian optimization to tune both the hyperparameters of the unsupervised model and the supervised model, but you are correct that they are being done in unison with the overall accuracy being the target. The lift you get from adding the unsupervised step (and tuning it) is quite substantial (and statistically significant).
The idea of tuning just the unsupervised part (or doing it independently) is great though. All the code for the post is available at https://github.com/sigopt/sigopt-examples/tree/master/unsupe.... It would be interesting to see if doing that would make for a better overall accuracy.
SigOpt isn't a constraint optimization package for solving things like k-SAT thought if that is what you were asking.
You could also try to bake this into the objective function (with an L2 penalty for how bad it violates some constraints), depending on how hard the constraints of the problem actually are.
L2 penalty would likely converge to an infeasible solution. Augmented Lagrangian would be better, but then you're making users handle dual updates. At that point I'd rather use an actual constrained optimization library that does the algorithm carefully, and use primal-dual interior point. Not having this kind of thing built in counts as "no constrained optimization" IMO.
[1]: https://github.com/Yelp/MOE
I say these things because as someone who is active in machine learning - I often want to optimize hyper parameters. The type of people that are serious about optimizing hyper parameters (i.e. people who may not like to use grid or random searches) for a model are usually some what technical. Your product seems to be catered to those who may not be too technical (very simple interface, etc). How will you balance what you expose in the future without giving away too much of your underlying algorithms?
SigOpt was designed to unlock the power of Bayesian optimization for anyone doing machine learning. We believe that you shouldn't need to be an expert and spend countless hours of administration to achieve great results for every model. We're wrapping an ensemble of the best Bayesian methods behind a simple interface [0] and constantly making improvements so that people can focus on designing features and their individual domain expertise, instead of needing to build and maintain their own hyperparameter optimization tools to see the benefit.
For experts who want to spend a lot of time and effort customizing, administering, updating, and maintaining a hyperparameter tuning solution I would recommend forking one of the open source packages out there like spearmint [1] or MOE [2] (disclaimer, I wrote MOE while working at Yelp).
We're currently the only active company offering it as a service.
While hyperparameter optimization is one of the most common use cases of SigOpt right now, the general Bayesian Optimization As A Service we provide has also been used to tune simulations and even manufacturing and process engineering [1].