*any continuous function
If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions?
Pretty much, but you have to show that neural networks can create a basis in that vector space which is essentially the proof presented in the article.
> If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation.
Differentiating to obtain a linear approximation does not give you an arbitrarily good approximation like the theorem does.
> A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions?
Neural networks using sigmoid transfer functions do not compute linear transformations anymore.
Importantly this theorem also states that you can approximate any function with only two hidden layers. A similar proof could not be made for a single hidden layer so it seems that the non-linearity of a single layer is not enough to form a basis for all continuous functions.
> A neural network computes linear transformations
They compute a nonlinear function of an affine transform. Neural networks are nonlinear functions.
The standard proof uses some functional analysis techniques but nothing too complicated to show you can get arbitrarily close to any continuous function with an NN. That includes things like step functions whose derivatives are not defined everywhere.
The set of all (real) functions is also linear, but is a lot trickier to work with.
If you are using gradient descent, then you'll need the desired loss function to be differentiable with respect to the parameters, but that's a totally different matter.
In practice, however, the input to neural networks is represented by floating point values, which is a discrete set. So pick whatever arbitrary function you would like, there is some continuous approximation to that function which is actually equal to it on every floating point value, and that function can be approximated arbitrarily closely by a neural network.