You're talking about training. That's a totally different question than approximation and representability.

If that was the reason, the argument would demand the function be differentiable, not just continuous, wouldn't it?

Piecewise continuous functions can be approximated by differentiable ones. You lose some properties though. As you pointed out the derivatives might not converge, or even exist, and the limit converges pointwise at best, not uniformly.

Do neural nets need to be continuous functions of the inputs, or of the parameters? My recollection is that it only has to be the latter.