Is a single SGD layer a neural net? Is an image filter or an audio filter a neural net? Is matrix multiplication a neural net? This would strain the intended definition even farther than it's already been strained.

But all of those are differentiable programming, and rightly so because they're all pieces that you use and compose together to make interesting learning mechanisms, including the ones that we vaguely refer to as "deep learning" now.

I like the terminology. It's not about what the original long-abandoned motivation for the design was ("neural"). It's not about how gratuitously complex you can make it ("deep"). "Differentiable" is about how it works and how we design it.