Hacker News Reader

Comment by breckuh | Hacker News Reader

Are there any practical Pytorch examples? Say my network training time is 12 hours, I wonder how beneficial this would be for hyperparameter tuning over just simple grid/random search? Or would I instrument my network in a way to iterate over hyperparams faster than at every epoch/run?

View on HN

We have not yet released examples of interfaces with Pytorch. Maybe with moderate number of hyperparameters the benefit compared to random search will be moderate, whereas it will be very significant with high number of hyperparameters. It also depends on how parallel you are. In all cases we have a wide range of algorithms with a common interface, so that you can compare.

We also use it for direct training of the weights of a network in reinforcement learning, not only hyperparameters.

Can you elaborate on the benefit for a high number of hyper parameters?

For small numbers of hyperparameters, sometimes just random search is enough. This is not an absolute rule, sometimes with just 4 parameters random search miserably fails... just my rule of thumb, empirically, is that for hyperparameters in machine learning (this is certainly not the case in general) random search is often enough for 4 to 12 hyperparameters if the budget for hyperparameter search is ~100 trainings.

A fundamental problem is as the number of parameters increase the probability of sampling from the edge of the hypercube increases. You will then not effectively explore the parameter space. This might be some what alleviated by a concentrated multivariate normal, but I guess that has its own caveat.

If you instead have a sampling algorithm informed by the loss functions you avoid this problem. (You instead might have to worry about local minima.)

Reply on news.ycombinator.com