Hacker News Reader

Comment by meow_mix | Hacker News Reader

More parameters is very exciting! I wonder if anyone here has read about what the relationship between the number of parameters on the network and the size of the dataset used on the network? Wouldn't such a large number of parameters risk serious overfitting by giving the model the capacity to fit to exact samples?

View on HN

I am one of the authors of "Outrageously Large Neural Networks". Yes - overfitting is a problem. We employed dropout to combat overfitting. Even with dropout, we found that adding additional capacity provides diminishing returns once the capacity of the network exceeds the number of examples in the training data (see sec. 5.2). To demonstrate significant gains from really large networks, we had to use huge datasets, up to 100 billion words.

Impressive work, mate!

Does mixture-of-experts works well the other way around, as a way to minimize power and hardware in common sized problems ?

And would it work in low resolution networks, like BinaryConnect ?

A good rule of thumb is 1 sample per parameter. So if you have 100m parameters, you should have 100m samples.

Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lead to overfitting. Clearly there is a gap between out theoretical understanding and practice. Check out a recent paper that explores exactly this questions:

https://arxiv.org/abs/1611.03530

Reply on news.ycombinator.com