Hacker News Reader

Comment by parrt | Hacker News Reader

True, people use a grid search, but I am always very uncomfortable using things as black boxes. How does tree depth affect generality etc...? Effectively using a model means understanding your tools, in my view, but easy to get started w/o the math as you say!

View on HN

Besides having a basic understanding of what the parameters do (this is depth, this is learning rate, etc) I don't see what insights are to be gained. The optimal parameters depend on the particulars of your dataset, that's why everyone just does a search.

Maybe I am wrong, does this tutorial contain a derivation from the math that shows something like "if your data has these properties then you should set maximum depth to be this value and learning rate to be this value, and dropout to be that value"?

Complex ML models can behave very counterintuitively in response to simple hyperparameter changes, which is why it's more pragmatic to check a lot of combinations (e.g. grid search). CPU time is cheap, research time isn't.

For example look at this tutorial about a regularization hyperparameter:

https://medium.com/data-design/xgboost-hi-im-gamma-what-can-...

Id think this is much more useful than anything about the math. How much can you deduce thats described there from the math, isn't this all just figured out by playing around with it?

The main point of this article is really to explain how gradient boosting works and why. The math is really there to show what the algorithm looks like in its general form. The Discussion of parameters was really just a bit of motivation. Think of this as a good explanation of why it is performing gradient descent in function space. That tends to be very hard to explain.

I just disagree with the claim that understanding the mathematical machinery is necessary for tuning hyperparameters. I doubt it is even helpful.

Reply on news.ycombinator.com