Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.
You're talking about training. That's a totally different question than approximation and representability.
If that was the reason, the argument would demand the function be differentiable, not just continuous, wouldn't it?
Piecewise continuous functions can be approximated by differentiable ones. You lose some properties though. As you pointed out the derivatives might not converge, or even exist, and the limit converges pointwise at best, not uniformly.
"Piecewise continuous functions can be approximated by differentiable ones."
It depends on how you measure the distance between functions. If you are using the uniform norm (as is used in the universal approximation theorem) then this is false.
Do neural nets need to be continuous functions of the inputs, or of the parameters? My recollection is that it only has to be the latter.
Member of a Sobolev space enough? I think we're just reinventing finite difference and element methods here.