Triage is built for this: training and evaluating giant grids of models & hyperparameters using cross-validation. Similar to igel, it abstracts ML to config files and a CLI.
It's designed for use in a public policy context, so it works best with: - binary classification problems (ex: the evaluation module is designed for binary classification metrics) - problems that have a temporal component (the cross validation system makes some assumptions about this)