byt143

There's also this: https://www.youtube.com/playlist?list=PL7RZyOlq_Xnw3B5per3IU...

Sure!

Is this dual in any way to the information bottleneck hypothesis?

One usecase: Do you want to do interesting cutting edge modeling of disease propagation dynamics or other complex systems simulation? Then check this out: https://julialang.org/blog/2019/01/fluxdiffeq

Here's context : https://jontysinai.github.io/jekyll/update/2019/01/18/unders...

There's nothing else like it.

Also here's the julia equiv to dplyr: https://github.com/queryverse/Query.jl

> I think it has potential to be the language of choice for non-neural-network stats/ML (that's python for the foreseeable future).

I think it will overtake python for NNet stuff also: https://julialang.org/blog/2018/12/ml-language-compiler

I think it will happen in the medium term future:

https://juliacomputing.com/blog/2016/02/09/static-julia.html https://github.com/JuliaComputing/static-julia https://github.com/JuliaComputing/llvm-cbe

I don't think so(https://www.tensorflow.org/xla/developing_new_backend). And even if so, it's not relying on LLVM optimizations (for TPU) or the LLVM API. (which is why some of these had to be punted back up to the Julia optimizer for XLA, which was done in a third party package!).

The point is that Julia's design, type system and multiple dispatch facilitates writing dynamic yet highly optimized code for a variety of backends, even those requiring static semantics (unlike LLVM).

There is no way you can look at that paper (or the Flux ecosystem, or the prob programming languages or the SSA IR autodiff) and chalk up Julia's success to just LLVM.

Ah yes, that is true. However Julia is tackling a harder problem in that the speed lag is presumably due to optimizing for custom element and table types.

>Julia's secret sauce is LLVM

That most certainly is not true. See this for example: https://arxiv.org/pdf/1810.09868.pdf

https://github.com/queryverse/Query.jl Allows for dplyr syntax to work with any iterable and custom table types using traits...So I think it's already beating R data munging in the flexibility department.

Still missing some verbs, but these will be added.

Thanks for writing up your thoughts!

I find Julia's core design to be excellent for general purpose programming, better than python in fact since it essentially solves the expression problem with it's type system and multiple dispatch.

It's external program interop is also more pleasant than Python's :https://docs.julialang.org/en/v1/manual/running-external-pro...

Sure, it doesn't have the same general library ecosystem, but even that is being remedied for core areas like web programming: http://genieframework.com/ (a full MVC framework), https://github.com/JuliaGizmos/WebIO.jl (write front end code without javascript) and I'm particularly excited for https://github.com/Keno/julia-wasm, which will allow Julia programs to be compiled for the browser.

For any packages than are python only, it has excellent python interop using the pycall.jl package, which even allows users to write custom python classes in Julia.

With regards to numerical programming, it's obviously already far ahead of swift, and IMO much better placed to beat it in the long run. For example the WIP zyogte package is able to hook into Julia's compiler to zero overhead diff arbitrary code. Using Cassette.jl, package authors can write custom compiler passes outside the main repo and in pure Julia: https://julialang.org/blog/2018/12/ml-language-compiler

In addition, it's macro system, introspection, dynamic typing and value types through abstract typing approach allows for natural development of advanced probabilistic programming languages: https://github.com/TuringLang/Turing.jl, https://github.com/probcomp/Gen, https://github.com/zenna/Omega.jl/pulse

How does this relate to previous work that's described NNets in the limit of infinite width as GPs?

Gotcha, thanks.

Can you elaborate on your claim that python 4 will probably ship with a JIT?

Sure, but Julia doesn't need it: https://julialang.org/blog/2018/12/ml-language-compiler

From the blogpost

"Automatic Batching To get the most from these accelerators – which can have significant overheads per kernel launch, but scale very well over input size – it is common to batch programs, applying the forwards and backwards passes to multiple training examples at once. In simple cases, such as with convolutional nets, it’s simple to handle this by concatenating, say, 10 images along an extra batch dimension. But this task becomes much harder when dealing with variably-structured inputs, such as trees or graphs.

Most researchers address this by taking on the significant burden of batching code by hand. Different solutions have been proposed for different frameworks (DyNet, TensorFlow Fold, which heuristically try to batch some high level operations together when possible, but these typically either have their own usability issues or do not achieve the performance of hand-written code.

We suggest that this problem is identical to that of Single Program Multiple Data (SPMD) programming, which has been well-studied by the language and compiler community for decades, and becomes visible in more recent approaches to batching like matchbox. Indeed, it is very similar to the model of parallelism used by GPUs internally, and has been implemented as a compiler transform for the SIMD units of CPUs. Taking inspiration from this work, we are implementing the same transform in Julia to provide SPMD programming both for scalar SIMD units and for model-level batching. This allows us to reach the ideal of writing simple code that operates on individual samples, while still getting the best performance on modern hardware."

Still a very restricted subset of python, can't autodiff custom types etc

Two of your issues are currently being worked on. The autodiff tracker stuff is temporary until the lower overhead and almost invisible compiler based AD mentioned in the blog post is fully ready. No custom types needed.

There are also various autobatching packages being developed.

Regarding the GPU semantics, wouldn't that be solved by simply using a distributed array of GPU arrays?

Since flux is lightweight, generic, modular and pure Julia, these things can be developed in third party packages.

Except much faster and much more generic. Eager execution is very limited in the subset of python it can differentiate, for example. There's more info in the OP blog post.

I believe Julia has generic kernels for many linear algebra ops (like matrix multiply) that the compiler can use to generate code for different backends and types as a fallback to BLAS.

Check out Makie https://github.com/JuliaPlots/Makie.jl

And the cool thing is, Julia's wonderful generic programming facilities mentioned in the blogpost are used to make Makie generic to backend (GL, WebGL, cairo etc). Relies on this library https://github.com/JuliaPlots/AbstractPlotting.jl

Here's a cairo version of makie https://github.com/JuliaPlots/CairoMakie.jl

AD arbitrary julia code with mutation, closures, built in and user defined functions, fancy data structures etc (given the algorithm itself can be differentiated).

Better support for static compilation Is on the roadmap

Julia has it's own "*verse" type data pipeline framework, with an even greater variety of backends and plotting solutions than R.

It's still in development (mutate and select have PRs) but it's almost there.

https://github.com/queryverse/Query.jl

Here is how to do explicit SIMD in julia: https://github.com/eschnett/SIMD.jl

Actually Julia is aggressively inlined because it's easy to statically determine function calls at compile time...that's why the libraries are able to use such nice abstractions.

It's already ahead of time compilable, just needs some work to emit smaller binaries and cover some corner cases

>Your code does tend to end up a bit c-like in those performance critical sections, but that's hardly surprising.

That isn't true. Closures, higher order functions and fused broadcast array expressions are all very fast except in some corner cases.

Yes you can.

Also that exact C example can be written almost verbatim in Julia, so C isn't "better suited" for even these small kernels, because one can drop to low level with even clearer syntax in julia