Hacker News Reader

Comment by syntaxing | Hacker News Reader

The author mentions he defines over fit as “Test error is always larger than training error”. Is there an algorithm or model where that’s not the case?

View on HN

Yeah, that's a crummy definition. You can easily force "Test error is always larger than training error" for any model type.

It's not even about "forcing". This is such common and expected behavior that it's surprising (and suspicious) when it isn't the case.

yes, that's better said.

A pedantic example: a model that ignored the training data would do just as well on the training set as on the test set.

If you do not train on the training set, then there is no training set, and your example is degenerate.

We've trained on the training set, but our fit() implementation was, "accept the input, do nothing with it, and then return." When we then evaluate the model on the test set and training set, assuming there isn't a large covariant shift between the two or something, we'll get approximately equal values.

So it's a degenerate case, but not of the training set. (And presumably that's partly what they meant by "pedantic".)

You can see regularly in practice where aggressive data augmentation is used, which obviously is only used on training data. But, of course, you'd still be 'overfit' if you fed in unaugmented training data.

I think they mean "always (statistically) significantly larger". They're probably imagining that something like cross-validation would make test errors approximately equal to training errors, but if you consistently see significantly larger errors, then you've overfit.

When you find the minima of your validation curve, rarely if ever is your test loss lower than your training loss. I don't think this necessarily means you're overfitting.

Under the Probably Approximately Correct framework, there's an x% chance your error on the empirical distribution deviates more than \epsilon away in either direction from that of the actual distribution, so one can assume similar things between two empirical distributions.

Yes and no. Suppose for example you give MNIST to a SVM and fit the model, then test it only on 0 and 1 digits, which are generally well discriminized, you'll get almost 100% accuracy in test, whereas 97% or less in training. (ye probably need some preprocessing, like using PacMAP or UMAP or whatever but the point is the same)

However, that's just because I decided the right data to test it onto. So, you can't really say much on a model using that definition.

Reply on news.ycombinator.com