Hacker News Reader

Comment by nonbel | Hacker News Reader

>"The problem is that understanding all of the mathematical machinery is tricky and, unfortunately, these details are needed to tune the hyper-parameters."

You don't need to understand anything about the math to run a random, or grid, or bayesian optimization, or whatever search of the hyperparameter space.

View on HN

True, people use a grid search, but I am always very uncomfortable using things as black boxes. How does tree depth affect generality etc...? Effectively using a model means understanding your tools, in my view, but easy to get started w/o the math as you say!

Besides having a basic understanding of what the parameters do (this is depth, this is learning rate, etc) I don't see what insights are to be gained. The optimal parameters depend on the particulars of your dataset, that's why everyone just does a search.

Maybe I am wrong, does this tutorial contain a derivation from the math that shows something like "if your data has these properties then you should set maximum depth to be this value and learning rate to be this value, and dropout to be that value"?

For example look at this tutorial about a regularization hyperparameter:

https://medium.com/data-design/xgboost-hi-im-gamma-what-can-...

Id think this is much more useful than anything about the math. How much can you deduce thats described there from the math, isn't this all just figured out by playing around with it?

The main point of this article is really to explain how gradient boosting works and why. The math is really there to show what the algorithm looks like in its general form. The Discussion of parameters was really just a bit of motivation. Think of this as a good explanation of why it is performing gradient descent in function space. That tends to be very hard to explain.

I just disagree with the claim that understanding the mathematical machinery is necessary for tuning hyperparameters. I doubt it is even helpful.

Complex ML models can behave very counterintuitively in response to simple hyperparameter changes, which is why it's more pragmatic to check a lot of combinations (e.g. grid search). CPU time is cheap, research time isn't.

Ahh it’s because of people like you that I’ll always have a job. He’s right folks Please don’t try to understand the math behind it, actually the less you know the better.

Please don't get personal or swipey in HN comments. Rather, the idea is: if you have a substantive point to make, make it thoughtfully; if you don't, please don't comment until you do.

https://news.ycombinator.com/newsguidelines.html

Where did I say not to try to understand the math behind it? I say its not required to tune hyperparameters.

To me, its doubtful that someone whose first impulse is to generate strawman arguments is doing a good job in analyzing data. Thats the basis for NHST, which ml tools like this are trying to get away from...

Reply on news.ycombinator.com