If you take linear neurons, then the whole network is just some linear (or affine function) of its inputs, and hence the universal approximation fails (not every continuous function can be uniformly approximated by linear functions).
No comments yet.