Julia 1.5 Highlights
julialang.org
julialang.org
Anyways, big props to the developers and contributors, even if this is just a "minor" release. I'm extraordinarily excited for the future of Julia and hope it continues to grow and be as awesome of a community as it is! :)
There are a few common issues: 1) You measure compilation plus execution instead of just execution. 2) Your code relies on global values that limit optimizations. 3) Your code doesn’t allow the compiler to determine all the types. 4) Your code forces allocations for intermediate values.
Especially 4) makes a big difference when you’re coming from NumPy or MATLAB. If you have a chain of vectorized operations (e.g. “X = α * A + β * B + γ * C”), the performance is quite similar between the languages, but in Julia you can enforce that these happen in-place with a single fused loop and zero allocations (e.g. adding “@.” in front). This can often give you another ~2x–10x speedup and make the difference between “similar to NumPy/MATLAB” and “similar to C/Fortran”.
Say you have a database, and need pull data, do some heavy math computation, then present this data in a pdf report with a QR code, pulling images and processing them in the report.
In Python, it is without bells and whistles: psycopg2, numpy/pandas, reportlab and may be use PIL. Wanna stick this on S3? boto3.
In Julia, it's all very fragmented. LibPQ is immature, DataFrames.jl is nice, ??? (what pdf conversion tool?), images.jl (300 stars), qr code generator?
The problem is not the speed with most tasks. The problem is availability of tools. I am sure that will come with it, but why not just use Python at that point? I am not a fan of Julia and that's nothing to do with its features. It has immature ecosystem, with terrible IDE support (Atom/Juno raises my blood pressure), debugging is painful if non existent, error messages are all over the place, everything falls apart as the immature dependencies change and error messages don't help at all.
Julia is fun in your jupyter notebook. If you try to build apps in production environment in my team, expect push back if not straight up refusal to initiate such a project in the first place.
Syntax was amazing around 0.4v and it went downhill from there.
I am sure someone is going to nitpick my comment and provide a way to do it in Julia, but that's missing the point. The point is Python is miles ahead of what it does. In production systems, robustness + maturity matters.
Also, don't forget ancillary aspects of a programming language. When we put a python repo together, I am rewarded by an endless supply of developers that I can hire and immediately work on it. With Julia, the supply of engineers is limited and it is such a pain to train people to use it, learn its quirks, spend nights and weekends fighting with it and the business doesn't give a fuck about it.
julia> @time using Debugger
0.115907 seconds (202.08 k allocations: 13.215 MiB)
julia> @time @run sin(1)
1.438537 seconds (1.85 M allocations: 90.376 MiB, 0.80% gc time)
0.8414709848078965
julia> @time @run sin(1)
0.003900 seconds (23.39 k allocations: 1.057 MiB)
0.8414709848078965But it's worth noting that CSV.jl is a file that was entirely re-written in the past 6 months and has probably the fastest performance of any CSV parser currently used. It was written using pure julia and takes advantage of tons of Base-julia features as well as multi-purpose packages.
Julia's design makes it incredible easy to write the kinds of tools you want in a flexible way.
EDIT: The parent post was edited somewhat (which means this comment and some siblings appear slightly out of place) and I heavily disagree with a lot of the issues raised. I will keep the current comment as is for the sake of completeness, but I think the parent's approach to criticizing Julia is somewhat disappointing.
> In Julia, it's all very fragmented. LibPQ is immature, DataFrames.jl is nice, ??? (what pdf conversion tool?), images.jl (300 stars), qr code generator?
A quick google search yielded this [0], [1] which seems like they'd do the job..
Personally I think judging the quality of the ecosystem on the number of stars a package has is short-sighted and gives only a skin-deep view.
> error messages are all over the place, everything falls apart as the immature dependencies change and error messages don't help at all.
When was the last time you used Julia? Because I've found the error messages personally far superior to Pythons errors, even in deep stack traces I find it easy to pinpoint the exact bit of functionality that has failed, why and understand what to do to fix it.
> I am sure someone is going to nitpick my comment and provide a way to do it in Julia, but that's missing the point. The point is Python is miles ahead of what it does. In production systems, robustness + maturity matters.
Python is mature _for now_, but this gap is gradually closing, and I've come across more than my fair-share of python packages that are abandoned, heavily reliant on magic, poorly written etc. The size of the ecosystem is not necessarily representative of it's value or worth.
> When we put a python repo together, I am rewarded by an endless supply of developers that I can hire and immediately work on it
This feels to me like the equivalent of "I'm looking for _React_ JS devs" - the knowledge and competency you have as a programmer should be able to be applied to new frameworks and languages, and I'd personally gladly hire someone who says "I'll learn the languages and tools I need to in order to apply my skills" instead of someone who says "oh it's not language + framework I won't/can't apply my skills", because the former is almost certainly going to be a much better programmer.
[0] https://stackoverflow.com/questions/53074626/how-to-generate... [1] https://github.com/jverzani/Mustache.jl
One area Julia could pull ahead in is
> It's amazing to realize that writing down the first thing that comes to your head is usually like 80% as fast as a good, performant implementation. (As someone who has done a decent amount of work in performance engineering for embedded platforms, I really enjoy squeezing out the last drop of performance from most programs, but doing this for every single first-pass at a program, like in Python, is rather annoying if this is what is needed to get a usable implementation.)
Yes, Julia is not necessarily faster than a good C implementation (that doesn't leak, etc), but, like Python, what would be a 300-line C implementation, where one has to somewhat carefully manage typing and the abstraction is really rather complicated for something that is mathematically simple, we can usually write 20 lines of very performant Julia that is 95% as fast.
Attempting to do something relatively similar in Python is often slow enough that giving a practical implementation essentially needs to be coded in C and interfaced with Python, where we return (again!) back to the same problem we had before: writing a 300+ line C file for something that should be rather simple, mathematically speaking.
Now that we're developing in Julia we can realize our designs as we envisioned them with very little code. Features like multiple dispatch have been a lifesaver for us. And we are getting performance that is quite close to C++. Now we're looking forward to using features like composable multithreading.
One could wonder "why not use python?" but for performance reasons one ends up with the 'two-language problem' which we wanted to avoid.
Other than Mir, there is very little to choose from.
D users put the language into a pedestal of language design, but that isn't what grows an eco-system, getting new users and libraries does.
I used to love the language, but so many mistakes have been made during the last 10 years, that it will hardly recover unless some company champions it, Swift/Kotlin style.
So after reading the release notes about 6 sec. of start-up times and people keep complaining about warm-ups, I am smirking about how Julia is a dynamic language when compiling & executing a script written in a static language other than let's say Scala/Rust/C++ can be comparable or even faster in some situations.
The warm-up period is definitely an annoyance, but a surprisingly small one. Even if it takes one minute to compile all the libraries and code I'm working on, my programming session is usually much longer than a few minutes, so that warm-up becomes insignificant as I keep the program alive during all the development process and any new addition are pretty much instantaneously compiled (unlike static languages that have to be frequently recompiled, and it's faster even compared to incremental compilation in languages like Scala) and at the same time running faster after warm-up saves time over the session compared to interpreted language as well. Never bothered with PackageCompiler.
It's a matter of different workflows, and since Julia isn't the same as the usual dynamic languages or the usual compiled languages, it's easy to end up with suboptimal ones especially at the start (which I assume does hurt the image of the language as first impressions are key). That said I'd definitely want the ability of creating small static binaries for deployment or end users (even if they don't help during development, which I'm already more than satisfied).
It was great to see so many JuliaCon talks last week using these GPU capabilities.
For reference, the maintainer of AMDGPU.jl can't upgrade to a new AMD GPU or he won't be able to use ROCm.
What am I missing?
FWIW, it is quite significantly improved in the upcoming 1.6 release.
1.5:
julia> @time using Plots
8.859293 seconds (15.99 M allocations: 913.783 MiB, 3.86% gc time)
julia> @time using DifferentialEquations
25.394033 seconds (58.78 M allocations: 3.222 GiB, 3.96% gc time)
julia> @time using Plots
3.957724 seconds (7.67 M allocations: 537.424 MiB, 5.50% gc time)
julia> @time using DifferentialEquations
9.708388 seconds (23.08 M allocations: 1.537 GiB, 6.20% gc time)A fundamental idea in Julia is Multiple Dispatch; methods are extended many times with different types, and common "verbs" (such as `open()`, `write()`, etc...) are used to operate upon a wide variety of objects (files, sockets, plot handles, sound devices, etc...). Exactly which pieces of code gets called when you call `open(foo)` depends not only on the type of `foo`, but also on what methods have been defined for `open()`.
As a concrete example, the deep learning library `Flux.jl` has code within it that checks to see if CUDA packages are already loaded or not within it. If they have been loaded, then `Flux.jl` will load its GPU support libraries as well. This means that:
``` using CUDAnative using Flux ```
and
``` using Flux using CUDAnative ```
Result in different environments, and hence, different code being run. It is unfeasible to precache all packages (you would, in the worst case, end up with a combinatorial explosion of packages that is `N!` in number of distinct package-package version tuples) and intelligent ways of thinning that down would quickly become untenable due to the highly dynamic nature of Julia.
Really what this is working around is the lack of support in the Julia package ecosystem for a way for packages to coordinate behavior; with stronger systems in place for a package environment to say "Hey, this is a GPU-using environment, and all packages should turn on GPU support!" there wouldn't be a need for this kind of dynamic behavior.
This is an example of something that happens all the time in the Julia world; when we lack a proper structure, users hack their way out of it with crazy dynamic code that does things that the language supports. We don't want to take that dynamic power away because that's where our edge over many other languages comes from, but it does make it challenging for the compiler to support everything in as low-latency a manner as a less-dynamic language would allow.
There are plenty of possible workflow improvements, of course, such as tighter PackageCompiler.jl and Revise.jl integration that would allow you to bake system images per-environment while still picking up some code changes, etc... but the penalty paid is pretty high; right now, building a system image takes many minutes, and it would be done for any kind of code change. There is compiler work underway to allow for more incremental system image compilation as well, but as with many things on our todo list, it must find its way into the compiler team's pipeline alongside all of the other cool things that are being worked on.
As mentioned in other comments, even without changing the workflow of Julia projects at all, incredible strides have been made over the last few releases, but I wanted to give you an idea of why certain things are harder than they might seem, when coming from other languages. As an aside, `Plots.jl` and `DifferentialEquations.jl` are two notoriously slow-to-load packages, beaten out only by other behemoth packages such as `Flux.jl` or `Turing.jl`.
Julia does the same things, faster and better, and literary with the same syntax.
That being said, python with numpy has got really fast. If you stay within numpy, and are careful with loops, or if numba works for your code, don’t expect much improvement.
Our customers use Matlab, our colleagues do, all our code, built up over years, is Matlab. Learning a new language incurs a cost, with somewhat unknown long-term payoff.
Julia is, in my opinion, far superior to Matlab (and also better than Python), but I'm stuck with Matlab (and a bit of Python) because of those things.
I didn't know why my code using views allocate memory, so I rewrote all my performance critical functions to return iterators (which is not that bad, as Rust uses lots of iterators, just Julia standard library is not optimized for them). At least now I understand what was the issue.
Regarding thread-safety I started writing a borrow-checker macro as a hobby, as I have been bitten by accessing mutable data structures from multiple threads. It's great that the macro and type system is so flexible that it can be done without modifying the language.
Kudos to everyone who contributed to yet another great release!
I'm a little nervous about code in the REPL and code in a file behaving differently. I think they should go back to the 0.x scoping rules everywhere consistently - which was lexical scoping, IIRC.
Which brings up a question: if this behavior is context dependent (different in REPL and in a file) does that mean you can set a flag so that code in a file behaves like code in the REPL?
> Keep the 1.0 behavior in files, to avoid breaking anybody's code.
> Print a warning if code in a file would behave differently than in the REPL, requesting an explicit local or global declaration to disambiguate the variable in the loop.
> An important property of this design is that any code that executes in a file without a warning will behave the same way in a fresh REPL. And on the flip side, if you take a REPL session and save it to file, if it behaves differently than it did in the REPL, then you will get a warning.
So there's no real danger of deviating behavior unless you're in the habit of ignoring warnings.
As to why bringing back 0.x scope rules is not a good idea, this post explains why it was changed in the first place: https://discourse.julialang.org/t/explain-scoping-confusion-.... In short, for non-toy programs, it's a significant source of hard-to-find bugs.
Also: changing this behavior in non-interactive contexts would violate the semantic versioning compatibility commitment of 1.x releases. Of course, 1.5 does sometimes introduce a warning in existing code, but (a) it will keep working the same way it did only with a warning and (b) this only happens if you had a global implicitly shadowed by a local, in which case there's a large chance your code was actually broken.
Give it a try. A lot of time and thought was put into this change and we believe the new design is pretty much locally optimal. It preserves the safety of the 1.0 behavior for non-interactive programming while recovering the convenience of the 0.x behavior in the REPL.
It seems like a lot of languages have lexical scoping which IIRC was what 0.x had. (Scheme, Lisp, OCaml are a few examples) Are there aspects of Julia that make lexical scoping particularly problematic? Julia 1.x scoping rules look similar to Tcl's.
Now, Julia has three distinct scoping algorithms.
See the three distinct behavior for the same two pieces of code:
Program Ⓐ:
l = 0
for i = 1:10
l = i
end
println(l)
local l = 0
for i = 1:10
l = i
end
println(l)
• TOPLEVEL: Program Ⓐ gives “0”, Program Ⓑ gives “UndefVarError: l not defined”.
• FUNCTION: Program Ⓐ gives “10”, Program Ⓑ gives “10”.
Warning: Assignment to `l` in soft scope is ambiguous because a global variable by the same name exists: `l` will be treated as a new local. Disambiguate by using `local l` to suppress this warning or `global l` to assign to the existing global variable.
And while you write "same piece of code", the code you then paste is not the same: the statement "local" appears in a different statement block, leading to it being executed in a subsequent scope block.
To be clear, I'm not saying the Julia devs have been ignoring runtime performance, far from it, but I just sometimes feel the need to chime in like this because I often feel the compiler latency complainers are disproportionately squeaky wheels, so I'd hate to see that cause the devs to over-correct.
When it comes to explicit trade-offs, like setting the optimization level, they've given us finer control by letting us specify it at the module level. Now Plots.jl can favor compile time, while I'll numerical libraries can favor runtime.
I'm just saying that for selfish reasons, I'd rather they spent their time on runtime and infrastructure improvements than reducing compile times.
And of course, I say this while acknowledging that I am in no position to demand how they spend their time.
Dot product: NumPy 0.03528 vs 0.03 Julia (x1/1.1)
Element-wise sum: NumPy 0.00379 vs 0.0063 Julia (x1.6)
Element-wise mul: NumPy 0.00419 vs 0.00617 Julia (x1.5)
L2 norm: NumPy 0.02391 vs 0.097 Julia (x4.1)
Matrix product: NumPy 0.00186 vs 0.01988 Julia (x10.7)
Matrix sort: NumPy 0.01033 vs 0.0161 Julia (x1.6)Also, it's unlikely that Julia will handily beat the handwritten, carefully optimized C-code in numpy. At that level, it isn't very much about the language but more about the programmers implementing the algorithms. A major selling point of Julia is that you will have fast code even in cases where the code doesn't neatly factor into calling static kernels written in e.g. C.
At this stage I can only see that it is the ecosystem which is hard to evaluate without investing into learning the language. However, I find it hard to believe that it is richer than Java or Python.
Notice how none of those languages have a vibrant numerical computing ecosystem anywhere near Julia's despite being just as old or older than julia and being big, well known languages with large communities? It's almost like scientific programmers don't want to do science in those languages.
I don't think it's an accident that Julia's package developers have been remarkably productive compared to Python. The best differnetial equations library available on Python right now is a wrapper for Julia's DifferentialEquations.jl. This is not because Python users don't care about differential equations or because there's not enough money or manpower. Chris Rackauckas did more on his own in julia for differential equation solving in his first couple years than entire industries did in Python over decades.
Julia is a productive and performant language. When you hold up Java and Python, you're asking me to choose between productivity and performance (at best, obviously Python has serious productivity problems and Java has serious performance problems despite those being their touted strengths).
Well, for that it depends on what you're into. For traditional, coding bootcamp machine learning, Flux.jl is still behind, but I'd point out that the only high performance, usable NeuralODE library is still built on Flux.jl and DifferentialEquations.jl as far as I understand and mostly consists of some import statements and a few function compositions / overloads. Someone already did deep learning on fully homomorphic encrypteded data using Flux.jl. Many 'niche', cutting edge things are still best done with Flux.jl because Flux is fundamentally more flexible than things like PyTorch and TensorFlow. It's using the entire programming language as a framework in a very generic way that PyTorch and Tensorflow can only ape by building ever more complex custom compilers and DSLs with ever diverging semantics from Python.
If Flux.jl is an example of the Julia community's failures to compete I'd say we have very little to worry about, and I look forward to seeing what a well funded commercial entity could build in Julia with a few billion dollars.
When it comes to work, niche languages lose their ground once you start digging deeper and attempt to venture slightly outside their focus domain. They do not have dedicated teams or industrial money and will always lag behind. Look at DL4J, they have a whole company working on it for many years.
Of course time will tell but considering the results so far I am not convinced that investing time into learning a new language will pay off. Especially considering Julia interop with C/C++.
Is the interop bad?
Well, running "Hello, World" still takes 0.3 ms :D
- Hello world is slow? Really? Yeah, it's not a hello-world-oriented programming language.
- One-based indexing is questionable? Troll much?
- Using `ccall` is unsafe? No kidding, there's no way to call a shared library that doesn't crash if you lie to it.
- `@printf` sucks? Then don't use it. I never do. The `print` function works for 99% of cases.
- Julia development is slowing down? Whiffed on that call.
Regarding the C ffi point, you can use Clang.jl to generate correct bindings so that you don't have to read C header files. But note that if you want to call a function in C you need to look at and match its signature, so it's not all that onerous to have to do the same from Julia. Julia's C interop is so good that several other languages have copied it since it came out.
You definitely should have because from what I saw during my google searches this idiotic article was mentioned a couple of times.
Flux is more generic, flexible and thus harder to build. Julia could have been there already if the small ML team w was working with TF type design goals. But they aren't, and therefore are ahead of TF/pytorch in many respects.
Also not as much effort has gone into making this polished for standard deep learning stuff, but you can see the fruit in things like reusing the same code for ML on holomophically encrypted data, faster than pytorch/tf deep learning on graphs and 3d inputs, faster, easier composability with ODEs and prob programming etc.
So any language on Windows able to use COM/UWP, gets those Python libraries, sorry, C++ libraries.
The effort to use Julia FFI to interoperate with them is hardly any different than using Python FFI, with the big difference that in what concerns Julia it would be possible to re-implement them in pure Julia, while the Python community keeps wondering about to write more FFI code to native libraries or finally start adopting PyPy.
The point of Julia is that Float64 and Int aren't special. They're on the same footing as whatever types you cook up, and our performant multiple dispatch system lets you build a vibrant interacting package ecosystem filled with user defined types doing targeted overloads of other functions.
It's the multiple dispatch and composability of julia code that matters. The important part of the performance of Julia is that we can get the ecosystem benefits of writing pure julia code without paying with runtime performance penalties.
My rule of thumb is that if your cherrypicked microbenchmark involves one specialized Numpy function call, it'll be fast, but if there is two separate specialized Numpy function calls it'll be slow.
julia> using LoopVectorization, BenchmarkTools
julia> A = rand(10_000, 10_000); B = rand(10_000, 10_000); C = similar(A);
julia> @benchmark vmapntt!(+, $C, $A, $B)
BenchmarkTools.Trial:
memory estimate: 13.19 KiB
allocs estimate: 91
--------------
minimum time: 29.103 ms (0.00% GC)
median time: 29.202 ms (0.00% GC)
mean time: 29.329 ms (0.00% GC)
maximum time: 30.658 ms (0.00% GC)
--------------
samples: 171
evals/sample: 1
>>> import timeit
>>> u = timeit.Timer("A + B", setup='import numpy as np; A = np.random.rand(10_000, 10_000); B = np.random.rand(10_000,10_000)')
>>> u.repeat(10, 1)
[0.17918606100283796, 0.17888473700440954, 0.17893354399711825, 0.1790916720055975, 0.17922663199715316, 0.17935074500564951, 0.17939399600436445, 0.17940548500337172, 0.17933111900492804, 0.17920518800383434]
Elementwise multiplication is the same fast.
If I get a little more creative and do exp(A) + log(B) instead, multithreaded Julia still takes 30ms, while single threaded slows down to 169ms.
Meanwhile, NumPy slowed down to 847ms, making it 30x slower than multithreaded Julia (on my computer) and 5x slower than single threaded.
Shall I keep making programs more complicated?
I made the arrays huge to get roughly the same ball park of times you reported. I also preallocated the memory in that benchmark to avoid unnecessary allocations (the observed allocations are because multithreading in Julia allocates memory). Is this something as easy to do in Python?
I made a few tweaks to your benchmark code. I get with Julia:
Element-wise sum of two 100x100 matrices (int), (1000 loops)
1.698 ms (2 allocations: 78.20 KiB)
Element-wise multiplication of two 100x100 matrices (float64), (1000 loops)
1.786 ms (2 allocations: 78.20 KiB)
Dot (scalar) product of two 300000 arrays (float64), (1000 loops)
4.801 ms (0 allocations: 0 bytes)
Matrix product of 500x600 and 600x500 matrices (float64)
229.475 μs (0 allocations: 0 bytes)
L2 norm of 500x600 matrix (float64), (1000 loops)
6.566 ms (0 allocations: 0 bytes)
Sort of 500x600 matrix (float64)
523.243 μs (595 allocations: 2.32 MiB)
| Element-wise sum of two 100x100 matrices (int), (1000 loops) | 0.0042680892984208185 |
| Element-wise multiplication of two 100x100 matrices (float64), (1000 loops) | 0.0027007726486772297 |
| Dot (scalar) product of two 300000 arrays (float64), (1000 loops) | 0.0088505959516624 |
| Matrix product of 500x600 and 600x500 matrices (float64) | 0.00107754109922098 |
| L2 norm of 500x600 matrix (float64), (1000 loops) | 0.010691180499998154 |
| Sort of 500x600 matrix (float64) | 0.008244920199649642 |
1. Small int matrix element-wise sum: 2.5x
2. Small float matrix element-wise product: 1.5x
3. Dot product (scalar): 1.84x
4. Matrix multiplication: 4.7x
5. L2 norm: 1.6x
6. Sorting: 15.8x
But I left all the inputs the same.
My primary point with Julia here is that it's easy to optimize it and get more performance out of it if or whenever you need it.
In the article, they said all the packages would be in tarballs and in their Pkg server. I haven't explored this yet but what I'm hoping to do is download all the packages from the links in the package registry, and then point the Pkg client to my local drive somehow.
The PkgServer protocol serves content-addressed chunks of data to Julia Pkg clients, so to download a certain version of a package, the Pkg client will request things like `GET /package/${pkg_uuid}/${content_hash}`. If you can pre-fill a PkgServer with all of the packages that you want, then it can serve a client just fine.
The full design is that there are a small number of "Storage Servers" that continually explore the global registry of packages (called `General`, located here: https://github.com/JuliaRegistries/General), downloading and storing tarballs for every version of every package that is available. These storage servers store everything forever, while Pkg servers contain an LRU cache to allow them to be deployed close to whatever compute resources will be requesting packages. For an airgapped solution, you could generate a static snapshot of some selection of the packages you want to serve, then serve them with nginx and point to that "static storage server" with the opensource Pkg Server, and it would all "just work".
An example of how to generate a static storage server is here: https://github.com/JuliaPackaging/PkgServer.jl/blob/master/b..., you would serve the resultant directory structure with something like nginx, then point the Pkg Server to that server as the upstream storage server.
I will note that Julia Computing offers an enterprise solution for dealing with secure/restrictive environments called JuliaTeam which provides this in a convenient, managed bundle, along with many other useful features.
EDIT: Ah, I forgot to mention, Stefan and I gave a talk at JuliaCon 2020 that touches on some of this, here's a link to the timestamp of the relevant section: https://youtu.be/xPhnJCAkI4k?t=350
And for more info, here's the original planning issue (note some things have changed as we've implemented it over the last year, but the bones are the same): https://github.com/JuliaLang/Pkg.jl/issues/1377#issue-492482...
On the contrary, random number generation has no general solution and remains an active area of research in many fields that depend on it.
julia> @which randn(1000)
randn(dims::Integer...) in Random at ~/Software/Julia/share/julia/stdlib/v1.5/Random/src/normal.jl:210
julia> @edit randn(1000)
# Opens the normal.jl file in your editor
julia> @run randn(1000)
# Starts the debugger in VSCode
[1] https://github.com/JuliaLang/julia/pull/29240 [2] https://arxiv.org/abs/1805.10941
Still have the impression that it's mostly used by academics or niche audience, would be happy to be proven wrong
If you prefer dynamic languages and interested or do scientific programming.
If you need your code to run LLVM fast.
If you have a lot of RAM and can live with "warm-up" lags.
If you are comfortable with MATLAB syntax.edit: for those who don't know what an ORM is https://stackoverflow.com/questions/1279613/what-is-an-orm-h...
I think it'd be cool to do some of the SQL generation stuff, but no one has sat down to design something out and how it would work in Julia.
This should not take more than 100 milliseconds in this day and age, folks.
Why is latency a concern for you? Are you running a lot of short, one-off scripts? If so, an easy solution is to just keep a julia session running and re-use it for each each script rather than constantly starting up and closing julia sessions. DaemonMode.jl [1] is a package that was recently developed that makes this workflow effortless.
Another option if you've got some production code that needs to be run over and over again is to create a custom sysimage with everything AOT compiler. PackageCompiler.jl [2] makes this workflow pretty easy nowadays.
(All the standard libraries are baked in by default, in much the same way I think, and so can't be updated without making a new version of Julia. But they all load fast.)
Here's the instructions: https://julialang.github.io/PackageCompiler.jl/dev/examples/...
It is actually exceedingly difficult due to multiple dispatch. There's an effectively infinite number of different method signatures a function can have.
Currently, anything you want to AOT needs to go into a big monolithic sysimage with the full julia runtime. We might be able to eventually do a shared library approach that you can dynamically link to, but it'll take time.
Doing this automatically without the user opting in would make latency problems worse, not better.
> Reading about this a bit more, it almost seems like whatever PackageCompiler.jl is doing could be automated and baked into the the core julia executable with simple options/flags.
Not really, no. Fundamentally, PackageCompiler is building a monolithic executable with your desired packages baked into it. Every time you want to add a new method to compile, you need to rebuild the whole thing and you cause it to be larger on the disk and slower to start up.
Even if we could quickly cache methods, I have hundreds of Julia packages installed locally on my machine. and regularly call methods with exotic signatures. If I baked every method I ever compiled into my sysimage, it'd probably be hundreds of terabytes in size at least. You have to remember that every time I call a function f on arguments x, y, and z, I need to compile a new method for each distinct signature
f(::typeof(x), ::typeof(y), ::typeof(z))
Of course, it's possible to do more caching and faster than we currently do, but I just want to emphasize that it's a hard problem.
jlpkg does this to speed up package installation: https://github.com/fredrikekre/jlpkg https://live.juliacon.org/talk/N39HSX
What is your point?
Then you have a stunning misunderstanding of Python's demographics.
If it’s number of installs, then Python tinkerers would win and I agree with you.
Sounds like a hard thing to measure, but I'm sure someone has good estimates.
> Also, if we define execution of a Python line as a metric: Instagram alone will dwarf all prototypers in Python. What would be a good metric to indicate “popularity”?
Code cycles isn't really what I would mean by language use, but if we go by that metric Julia's production code also dwarfs non-production code cycles.
The Celeste project alone was running at petaflops[1] at it's peak, that is 10^15 floating point operations per second. One of the biggest targets for Julia is deploying it at scale on super-computing clusters. There are almost surely many more CPU cycles being devoted to production scale julia code than random repl code, but that's got to be true for almost any language being used at scale.
Also, counting Python line executions in production code is a little squirreley, because almost surely those lines are really just a thin wrapper around a big hunk of C code if it's a production system.
> If it’s number of installs, then Python tinkerers would win and I agree with you.
I'd also say just number lines written or projects started, the tinkerers win hands down.
[1] https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-cl...
Slimmer separate compilation is on the roadmap