I suppose it depends on the activation function, but many of them are bounded. Polynomials are unbounded at the extremes, and as such should really only be trusted (sometimes) for interpolation, not extrapolation. If you get a new input which is outside the radius of your training set, the answer could be pretty extreme and not supported by the training data.