Comment by YeGoblynQueenne | Hacker News Reader

Comment by YeGoblynQueenne | Hacker News Reader

Parent Full thread

YeGoblynQueenne

>> RNN's can be taught to do math using BP

The paper you cite and every single paper on artificial neural nets learning to "do maths" or "do arithmetic" etc that has every been published. are only showing neural nets learning [1] the results of specific operations between numbers up to a certain value. There is no work that shows neural nets learning to do arbitrary arithmetic operations on arbitrary numbers.

To put it very clearly: neural nets can't learn to "do maths" in any general sense.

Neural nets are infamously incapable of generalising beyond their training dataset and the OP is very reasonably skeptical of comparisons between artificial and natural neural networks, given that only the latter are known to be able to generalise from few examples seen very few times.

______________

[1] As in overfitting to.

>Neural nets are infamously incapable of generalising beyond their training dataset

In what sense? DNNs can certainly generalize to examples outside the training set. I agree that you get no guarantees on performance for samples drawn from a distribution that is different than the training/test set. Not having guaranteed expected performance isn't the same as "incapable" of generalizing to a new distribution.

Sorry for the late reply.

To clarify, when I say "training dataset" I mean the entire dataset used for training. Not the training partition in a cross-validation training/test split.

Neural nets can interpolate between the data points in their training dataset, but cannot extrapolate to cover data points outside this dataset. This is not a matter of architecture. DNNs are no exception. Neural nets are just very bad at generalising.

This lack of generalisation ability is why neural nets need to be trained with huge amounts of data, the more the better. Because they can't generalise, the only way to get them to recognise more instances of a class is to show them more examples of that class.

Here's a longer discussion of this:

https://blog.keras.io/the-limitations-of-deep-learning.html

Reply on news.ycombinator.com