Are there things that you've done with probabilistic programming that aren't sophisticated, but were still a good use of pp?
Or, in other words, what's a minimal example of a problem that is significantly simplified with probabilistic programming?
My intuition says that this stuff is great, but conceptually I can't think of use cases remotely relevant to a non-statistics specialist, i.e. me.
[0] https://camdavidsonpilon.github.io/Probabilistic-Programming...
Being able to run generic codes and packages from a whole language is really freeing because people in Stats don't generally have the same models as a systems biologist, pharmacometrician, computational fluid dynamics researcher, etc., so you know in advance that there will never be "built-in support" for any of these disciplines. And that's fine as long as the system is extensible. I like the Julia-based probabilistic programming frameworks because they let you put entire differential equation solvers in there, and estimate the parameters of some scientific model in a way where you get posterior distributions that quantify the uncertainty. Stan hard-coded 1-2 ODE solvers in there for "similar" functionality, but here we get a few hundred ODE, SDE, DDE, DAE, PDE, jump diffusion, etc. methods which all come along for the ride. Making something compatible with a PP framework like Turing.jl is essentially just making it compatible with the AD framework the PP language uses (for derivative-based sampling). I write the differential equation solvers, and we were able to get PP working without a change to the PP or DiffEq libraries, which is quite a win in my book.
This might not answer your question directly, but it is perhaps useful to know that the probabilistic programming approach can also be applied in non-probabilistic (non-statistical) ways. I developed a ranked programming language, which is like a probabilistic programming language without probabilities. Instead, you state how your program normally behaves and how it may exceptionally behave. Conceptually it's very similar to probabilistic programming, but the underlying uncertainty formalism is replaced with ranking theory, which works with integer-valued degrees of surprise.
You can find an implementation of this idea (based on Scheme/Racket) here:
https://github.com/tjitze/ranked-programming
For more detailed information check the paper linked to on that page.
This is because I prefer to use Bayesian modeling techniques when I can, and PP is the best and most expressive method for implementing and fitting Bayesian models that I know of.