So it's still just neural nets? Cool -- we've seen that before.
But all of those are differentiable programming, and rightly so because they're all pieces that you use and compose together to make interesting learning mechanisms, including the ones that we vaguely refer to as "deep learning" now.
I like the terminology. It's not about what the original long-abandoned motivation for the design was ("neural"). It's not about how gratuitously complex you can make it ("deep"). "Differentiable" is about how it works and how we design it.
Differentiable functions
It's almost like SGD just made decades of AI research into neural networks just vanish.
Nobody is claiming that the definition of "differentiable programming" should be identical to the definition of "neural net". The claim is, if you want to assign a name to the thing that TensorFlow, PyTorch, and similar frameworks do, it's "differentiable programming".
If you want to make a non-differentiable neural net, knock yourself out. The research still exists and nobody is stopping you.
But while we're talking about terminology, I'd encourage you to stop referring to the units as "neurons". The false analogy to biology just confuses people.