Hacker News Reader

Comment by moffkalast | Hacker News Reader

Well iirc a convenient trait of a random forest classifier is that it cannot overfit onto learning data. Something that's not exactly true for deep learning.

View on HN

Any reference for this claim? In my opinion this is most certainly not the case - random forests are hard to overfit compared to gradient boosted trees but you can overfit with them too if you don't tune your parameters right.

Overfitting is generally a function of the the size of your data and the complexity expressible in your model.

My reference would be my machine learning prof mentioning it at the lectures a few years ago, something about decorrelation as the other guy in the replies mentions. It's possible he meant hard to overfit in comparison to something, not completely impossible.

A random forest is an ensemble model - i.e. lots of instances of a model, all separately fitted to the data, with some randomness so they're all different (e.g. bootstrapping, random subset of the data used to train etc). These models are then all used to classify, and their results averaged. This averaging will work to mitigate any overfitting that a single model might suffer from.

If you keep adding more and more trees to a random forest, variance will go down (they stop being 'random' and become more like the average over all possible trees). In this respect they can't overfit, but in practice there are other tweaks you can make such as tree depth and number of features) that you often want to tweak to increase model performance. These can certainly cause orverfitting.

RFs can overfit but in practice usually won't.

I think Hastie & Tibshirani have an article on this.

I think the origin of the claim is from Leo Breiman himself [0]. I’m not sure if he discusses this in any papers, but probably.

Sibling comments have some nuanced discussion of overfitting in tree-based models

0: https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...

I'm curious on this claim. Feels like any model can over fit.

A model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit.

The thing about deep learning is that the number of parameters is astronomical.

Hence, the interesting question is why deep learning models don't overfit all the time.

This is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/

Agreed, but it doesn't really explain it in any real sense.

“However, in the over-parameterized regime, there are many models that fit the train set and there exist such good models.”

One explanation is that a very large model effectively fits several models and penalises over-fit models itself.

Work is ongoing, this is a whole field of research now, a good keyword to search for is "implicit regularization".

I really like how AI research is all about empirical analysis of mathematics.

The answer is really very simple: because we use training and test sets, and generally use iterative training methods. As soon as test set error plateaus or starts to degrade, you stop training. This is called early stopping and can be considered a form of regularization. In theory it would seem a neural network (with early stopping "correctly" implemented) can never overfit, although I suppose in practice it can vary a little (because of test variance, too large step sizes, etc.). It's possible to train huge overparametrized networks this way.

One big problem I can see with this approach (maybe compared to over forms of regularization) is that it might be very space/memory inefficient. But if you've got a big budget I think it's a surefire way to make sure your model performs as well as possible (without having to tune hyperparameters too much or deal with local minima that probably happens at near-capacity networks).

Random Forests can overfit, but with sufficient data (and no data leakage) it is difficult to do so.

The final output of a random forest regression for example is just the average of all the final prediction nodes of the individual trees, which thanks to bootstrapping are decorrelated. So - adding more trees does not tend to alter predictions very much.

With algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.

Random forests can overfit, but they naturally have fewer parameters than nets, so don't overfit as a function of training time.

Neural nets have far more parameters and so are susceptible to overfitting with more training time.

If you train your ensemble long enough, won't it overfit too?

No, it's not sequential. For a given number of bootstrap replicates you can over fit by using trees that are too complex, but you won't overfit in the number of bootstrap replicates like you would with Gradient Boosted Trees.

Can you elaborate? I am unfamiliar with gradient boosted trees.

I think the idea is that gbt's are trained sequentially on the "residuals" left from the previous tree, while rf's are uncorrelated (can be trained in parallel, don't depend on one another). This means that in rf's there's no "compounding" effect that can lead to overfitting. The main way to overfit rf's would be to train many large trees.

Reply on news.ycombinator.com