Comment by crdrost | Hacker News Reader

Comment by crdrost | Hacker News Reader

A very similar effect is harnessed in current machine learning models and is called "dropout". The basic idea in context is:

- you take your reference data and reserve a random chunk of it to figure out how good you're doing, this data is only used to grade performance -- the rest is for training.

- you notice that as you train on the training data, you start improving your performance on the evaluation data, but only up to a point: then as you train more, the neural net starts getting worse on the evaluation data.

- this problem is called "overfitting", your neural net is now trying so hard to get the "details" right that it is losing accuracy on the "big picture".

- so we create a circumstance where the neural net cannot properly do overfitting because those "details" cannot really be resolved, by just dropping nodes at random out of our neural network as we're training it. It will have to pick up a level of redundancy in the nodes and the associated noise of losing nodes should stop the later stages from really having a "focused view" of the training data, such that it can overfit that data.

If you're interested see e.g. Adrian's Medium post about it, here: https://medium.com/@bingobee01/a-review-of-dropout-as-applie...

Dropout is traditionally only applied at training time: c.f. "At test time the weights are multiplied by their probability of their associated units’ dropout." from the medium post you linked. I _suspect_ that leaving it on during test has been accidentally tried a bunch of times though :) - so probably is not effective, although it could be interesting to experiment more intentionally with test-time noise. Our brains do not separate so cleanly between train and test, and so it seems more natural for the noise to be always on (if that's even the case as it seems hard to know what's noise when you haven't completely decoded the brain's activity). Decoding the brain seems to be super promising though and I hope we get good enough so as to upload our minds someday. Here's an interesting related article I read recently towards that end: http://www.foldl.me/uploads/papers/ccn2018.pdf

Dropout is computationally expensive though. It does address overfitting (by doing something like training an ensemble of weaker models, none of which will individually overfit, but all of which capture different "views" of the dataset), but it's not without its drawbacks, and people seem to be using it less these days.

Qua? I was under the impression that it was just going into architecture with less remark - ubiquitous, rather than dying. It's a great step towards building sparse models, as well, which is important for client side deployments.

Overfitting occurs when details are modeled that aren't justified by the data. i.e. not statistically significant.

Dropping nodes seems a roundabout way of addressing this, and you'd get a similar effect by training less.

Reply on news.ycombinator.com