Hacker News Reader

Comment by JumpCrisscross | Hacker News Reader

>No matter what the function, there is guaranteed to be a neural network so that for every possible input, x, the value f(x) (or some close approximation) is output from the network

Love NNs, but accuracy is relative. At some resolution of "close approximation," every function "computes" every other function.

View on HN

What the author means by "close approximation" is clarified later (under the "Two caveats" header). The point is that you can get an arbitrarily close approximation of any continuous function, just by choosing a sufficiently large number of hidden units. That is, for any epsilon > 0, a neural network can approximate any function within epsilon of the function's exact value at all points, with enough hidden units. The smaller the epsilon, the better the approximation you want, and the more hidden units you need.

What if you're not approximating a continuous function?

I think it's mostly accurate in L2 norm and not point by point. So every function in L2 can be approximated arbitrarily in L2 norm.

This is not true. The theorem proves point-wise convergence for a continuous functions on a compact subset of R^n. See theorem 2, page 6: http://deeplearning.cs.cmu.edu/pdfs/Cybenko.pdf

Well, compact continuous functions are dense in L1, so if you're discontinuous but at least integrable, you probably will still get a nice approximation. Take a box function for instance. Strictly speaking this is not continuous, but it's easy to see it can be approximated arbitrarily well by a couple of sigmoids.

Reply on news.ycombinator.com