I should add that Gordon Plotkin gave a keynote at POPL this year, Some Principles of Differentiable Programming, where he discusses this from the angle of probabilistic programming. It's on Youtube at:

https://www.youtube.com/watch?v=qhPBfysSYI8

It's a really inspiring talk, albeit super intense (you need to be bilingual in denotational semantics and real analysis to follow everything).