OtterTune is something that we have been working on at Carnegie Mellon for several years now. Your overall assessment of our approach is correct. OtterTune uses Bayesian Optimization, either with a GP or DNN as the surrogate model to predicate how the objective function will change for a given set of knob values. We then run GD to find a new set of values.

A lot of the tricky parts in this problem are figuring out what to tune, how to tune, and when to tune.