Turing.jl: A Fresh Approach to Probabilistic Programming in Julia
turing.guru
turing.guru
JuliaCon 2017 | Turing: a Fresh Approach to Probabilistic Programming | Kai Xu
Both use Hamiltonian Monte Carlo. As far as I know STAN cannot model factors but can restrict a variable to integers. In this case, Turing.jl and STAN do the same and switch back to slower MCMC.
So why should I switch to turing.jl?
Their wiki shows that turing.jl is about 10 times slower than STAN:
Results
Model Turing Stan Ratio
Simple Gaussian Model 1.2 0.06 20.24
Bernoulli Model 1.53 0.05 32.73
School 8 2.34 0.1 24.41
Binormal (sampling from the prior) 0.88 0.11 8.37
Kid 32.66 4.72 6.92
LDA-vec 23.94 3.78 6.34
LDA 72.78 3.78 19.28
Mixture-of-Categorical 22.28 6.41 3.48
NOTE 1
Numbers here are inference time in second - smaller number indicates better performance.Incidentally, I haven't been able to get Turing to run on Julia 0.6, and am not going to reinstall an earlier version just for Turing. I've been waiting for a little while now.
My experience with Turing and MAMBA have kind of diminished my enthusiasm for Julia. Both libraries kind of represent what I was looking for in Julia (similar to what you mention about using your Julia stack), but the speed was a kind of a rude awakening. I'm kind of coming to the opinion that LLVM-based languages need to demonstrate much more consistent performance before they're really ready to replace C (Rust, Nim, and Crystal seem like they might be on their way, though).
I recommend someone thinking about Julia pay attention to these benchmarks:
https://github.com/kostya/benchmarks
Yes, Julia is really fast with some things, but for other things it's much slower, and those slow parts become the lowest common denominator.
Hopefully things improve, though, because I do like Turing and MAMBA on paper.
As for the benchmarks you posted, the worst I see Julia performing in that list is on the JSON benchmark. However, again, that's benchmarking a library. Doing high performance text parsing well is a non-trivial problem and JSON.jl isn't a super high performance library. The fact that nobody has sat down and written one yet is more of a reflection of the user community (since JSON is mostly associated with the web stack, JSON-parsing tends to be more incidental for julia users, than languages/libraries that are more focused in that area) than any performance limitation of the language - in fact I think I recently did see somebody working on a higher-performance version, but I don't have the link handy.
To summarize, it is true that some languages (or rather their implementations) are slow, causing an automatic slowdown to projects written therein, but that is generally not the case for julia (or C or Nim or Rust, etc). Once you get to that point, other things matter more, and it is important to distinguish that. In the case of Turing.jl, the linked wiki mentions that one of the primary factors impacting performance is their use of forward AD, when reverse AD would be algorithmically better. It seems understandable, that a newer project would be algorithmically less sophisticated than a more established one. However, that'll get fixed over time and the other benefits will become more and more prominent as a result.
No sane person would deny this :) Regarding those benchmarks, however, I would note that:
a) They are two major Julia versions old, many of the benchmarks Julia appears in are string-centric, and since then Julia's string handling has been improved.
b) Glancing through I see a few pretty basic ways in which the Julia code could be sped up without doing anything convoluted
I really like Julia, so I don't want to come across as a hostile critic, more of a friendly one. I'm someone who has, and continues to try, Julia for lots of things. But repeatedly I've bumped into this phenomenon where Julia is maybe 3-5x slower than what you'd see in C, maybe more, and am faced with what that means in practice. It's not just Turing, and it's not just a single version of Julia. It's a heck of a lot better than Python or R, but when I realize it's still significantly faster to do something in C, and you're going to wrap something around C anyway, maybe Python or R would be better just because of the resource base associated with them.
Maybe there's some fundamental issue I haven't grappled with, like Julia will never actually be as fast as C, and I just need to be aware of what that means, practically speaking... I don't know. Maybe it will eventually hit that performance range, and it's just a matter of optimizing algorithm choice and whatnot. But I get the sense that the way I was hoping to use Julia isn't actually doable, at least at the moment.
I do think many people are coming to Julia because of some sense they can get the performance of C with the expressivity of Python, and at the moment I think that's an unrealistic expectation for a lot of things. I think the parent post in this subthread exemplifies that quite well: for many, many reasons, it shouldn't be surprising that Stan outperforms Turing. If this was any other language, it would be a matter of course. But because of the expectations set up for Julia, you see discussions like this.
Poor communication.