Can someone informed give some suggestions as to compare/contrast to other tools at the intersection of probabilistic programming and deep learning? What are relative strengths and weaknesses vs edward or pyro?
Turing is a bit like Stan, JAGS, or BUGS, it's [closer to] a general probabilistic programming system with a Bayesian emphasis, although maybe less Bayesian in emphasis than those other DSLs. PyMC would be another comparison.
Edward is (or when I used it) coming more from a very general latent variable modeling framework, encompassing hidden variable models, and is less focused on Bayesian modeling per se, maybe more variational inferential approaches. It seems to be broadening in scope over time.
Pyro I think is further down the deep learning/NN path than Edward.
It's hard for me to ennumerate strengths/weaknesses, as they have different foci and are parts of different language ecosystems. It depends a bit on the use case. My own experience with each is such that I might use Turing, or move to something like TensorFlow; things like Edward or Pyro seem to occupy this intermediate ground that was difficult for me to utilize in the way I thought I might.
I've been excited by Turing, just to see a probabilistic programming framework like that in Julia. I think the expressiveness of Julia and it being native to that framework will be helpful.
I hope so too. But hasn't Julia's TF/Torch equivalent, Flux, had performance problems? That was the rumor I heard anyway, I haven't had the chance to use it myself.
This is slated to be fixed in the short term with an abstract tracing framework which eliminates memory allocations. Given Julia's type information and ability to manipulate IR from third party programs, this is more general and powerful than pytorch's tracing system. Works on a larger set of code (the whole language), doesn't require actually running the code (abstract tracing), and allows for other program transforms/ analysis like compilation to XLA, shape inference, compile time errors, source to source prob programming : https://github.com/MikeInnes/Poirot.jl, and other things.
That's in the short term and should bring flux up to SOTA for speed (it already is on CPU). In the medium term, a general framework for optimizer passes will allow for more general compile time memory management.
As with most things in Julia, the code developers don’t just want to hack changes that work, but make changes that are flexible, extensible, and can solve many problems at once. So, Flux isn’t ready for prime time yet, but it is definitely worth keeping your eye on it.
So while it's in some sense similar to PyMC3 or Stan, there's a huge difference in the effective functionality that you get by supporting a language-wide infrastructure vs the more traditional method of one-by-one adding features and documenting them. So while PyMC3 ran a Google Summer of Code to get some ODE support (https://docs.pymc.io/notebooks/ODE_API_introduction.html) and Stan has 2 built-in methods you're allowed to use (https://mc-stan.org/docs/2_19/stan-users-guide/ode-solver-ch...), with Julia you get all of DifferentialEquations.jl just because it exists (https://docs.sciml.ai/latest/). This means that Turing.jl doesn't document or doesn't have to document most of its features, but they exist due to composibility.
That's quite different from a "top down" approach to library support. This explains why Turing has been able to develop so fast as well, since it's developer community isn't just "the people who work on Turing", but it's pretty much the whole ecosystem of Julia. Its distributions are defined by Distributions.jl (https://github.com/JuliaStats/Distributions.jl), its parallelism is given by Julia's base parallelism work + everything around it like CuArrays.jl and KernelAbstractions.jl (https://github.com/JuliaGPU/KernelAbstractions.jl), derivatives come from 4 libraries, ODEs from etc. the list keeps going.
So bringing it back to deep learning, Turing currently has 4 modes for automatic differentiation (https://turing.ml/dev/docs/using-turing/autodiff), and thus supports any library that's compatible with those. It turns out that Flux.jl is compatible with them, so therefore Turing.jl can do Bayesian deep learning. In that sense it's like Edward or Pyro, but supporting "anything that AD's with Julia AD packages" (which soon will allow multi-AD overloads via ChainRules.jl) instead of "anything on TensorFlow graphs" or "anything compatible with PyTorch".
As for performance and robustness, I mentioned in a SciML ecosystem release today that our benchmarks pretty clearly show Turing.jl as being more robust than Stan while achieving about a 3x-5x speedup in ODE parameter estimation (https://sciml.ai/2020/05/09/ModelDiscovery.html). However, that's utilizing the fact that Turing.jl's composibility with packages gives it top notch support (I want to work with Stan developers so we can use our differential equation library with their samplers to better isolate differences and hopefully improve both PPLs, but for now we have what we have). If you isolate it down to just "Turing.jl itself", it has wins and losses against Stan (https://github.com/TuringLang/Turing.jl/wiki). That said, there's some benchmarks which indicate using the ReverseDiff AD backend will give about 2 orders of magnitude performance increases in many situations (https://github.com/TuringLang/Turing.jl/issues/1140, note that ThArrays is benchmarking PyTorch AD here) which would then probably tip the scales in Turing's favor. As for benchmarking against Pyro or Edward, it would probably just come down to benchmarking the AD implementations.
1. Create "diffeqcpp", a C++ interface to DifferentialEquations.jl (that would be similar to diffeqpy, diffeqr) possibly using CxxWrap.jl
2. Make it possible to evaluate vector-Jacobian products (VJP) with "diffeqcpp". Probably that would require ODE RHS to be coded as a string of Julia code, to make Julia AD libraries compatible with it.
At this point, it should be possible to call Julia solvers from C++ and evaluate the derivatives.
In Stan, there is stan::math::adj_jac_apply that makes it possible to define custom functions with custom VJP without having to deal with Stan autodiff types, it works for example with Eigen::Matrix<double>. https://discourse.mc-stan.org/t/adj-jac-apply/5163
3. Make a class (let's call it JuliaODESolver) that implements two methods:
operator() // calls Julia solver for the given input
multiply_adjoint_jacobian() // evaluates VJP for the given vector
return stan::math::adj_jac_apply<JuliaODESolver>(ode_solver_inputs);
5. Modify cmdstan/main.cpp to initialize and finalize Julia context to be able to call Julia functions. This is probably the only place where Stan source itself needs to be modified.
I don't know what would be needed to make forward mode, and higher-order derivatives to work.
I think it would be much better for a fair benchmarking if there was a convenient and documented interface to Stan algorithms to use with user-provided log-density function, similar to DynamicHMC.jl and AdvancedHMC.jl libraries. It would be then easy to call it from Julia/Python/R/C++ or anything else.