The problem is that the true, hidden function being modeled is non differentiable. We can try to approximate that hidden function with one that is differentiable, yes. But no good approximation is guaranteed to exist depending on the nature of the problem. An example from op:

What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable.

CNNs can work for image classification because a smooth gradient will exist for incrementally "stepping" color values during backprop. But asking the question "is this an image of a cat" is much different than "what do I do if this is an image of a cat".

Also very deep neural networks (i.e. many, many time steps) essentially lose their gradient, or context. Even with attention and transformers. Which is partly why we haven't seen AIs that can write lengthy programs, books etc that are coherent and grammatically correct. And we probably never will by only relying on current "curve fitting" techniques.