Taylor series only approximate an (infinitely differentiable) function well locally, i.e. in some neighborhood of the approximation point. There are other methods applicable to less smooth functions and which approximate well uniformly (bounding the maximum absolute error) or by some other metric (e.g. L2 distance — integrated squared error).

Fourier transforms also smell quite similar. With those you also know how many samples you need to represent a function which changes at some known maximum frequency, using the Nyquist theorem. Is there anything translatable with neural nets? Like would you be able to prove you need X nr of neurons to represent a given function?

It seems the choice of function inside the neuron is kindof arbitary or is it locked to the sigmoid function? If so you could put cos and i*sin as function in your neurons and your neural net essentially becomes DFT?

https://en.wikipedia.org/wiki/Discrete_Fourier_transform

Well, in the sense that polynomials are universal approximators too (on a compact domain). See this comment:

https://news.ycombinator.com/item?id=19709834

No.