Can someone please explain to me, a mere mortal, what is the big deal with Julia. Why use it, when there are so many other good languages out there with more community/support? Honest question.
1 https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-cl...
2 https://arstechnica.com/science/2020/10/the-unreasonable-eff...
(Not a hater of Julia at all, very much think it's a cool language and an increasingly vibrant ecosystem and have been consistently impressed when Julia devs have spoke at events I've attended)
In other words, you can (empirically) get a lot done that way, but there is always friction.
Differential equation solvers that need to take a very custom function that works on fancy R/Python objects is another example of clumsiness in these drop-to-C-for-speed languages. It works and as a performance-nerd I enjoy writing such code, but it is clumsy.
That type interoperability is trivial in Julia.
The only difficulty with Rcpp-based R packages is you have to ensure the target system can compile the code, which means having a suitable compiler available.
For instance, I imagine there is an R library that makes it easy to automatically run R code on a GPU. Can that library also work with Rcpp functions?
I am very surprised by this. Given how R is extremely dynamic. and has things like lazy-evaluation, that you can rewrite before it is called with substitute. Which I am sure some packages are using in scary and beautiful ways.
In my case I went to deploy on a musl system and things with the two language just were a pain to get up and running.
Conversely, everything that was native python ran fine in a musl based python container.
Your native python code just moves also nicely between windows / linux / etc
I've been impressed with Julia, but it's hard to beat 25 years of package development.
EDIT: And last time I checked, Numpy only parallelizes calls to supplied linear algebra routines, and only if you have the right library installed. A simple vector arithmetic operation like a + b will execute on one core only.
Most people use a ton of numpy and scipy. It turns out that phrasing things as array operations with numpy operators is quite natural in this field, including for things like galaxy merger simulations.
I work, in particular, on asteroid detection and orbit simulation, and it's all pretty much Python.
You can get very far with these approaches I python, but having these at the language level just has more potential for optimization and less friction.
The debugability of numba code is very limited and code coverage does ot work at all.
Having a high level language that has scientific use at its core is just great.
Python has the maturity and community size on its side, but Jul is catching up on that quickly.
The Julia-for-astronomy community is just microscopic right now, so it's hard to find useful libraries. Nothing comes close to, say, Astropy[0].
I'm not a huge fan of the current numpy stack for scientific code. I just don't think anyone should get too carried away and claim that Julia is taking the entire scientific world by storm. I don't know anyone in my department who has even looked at it seriously.
“It turns out that phrasing things as array operations with numpy operators is quite natural in this field”
But if A and B are numpy arrays, then A + B will calculate the elementwise sum on a single core only, correct? It will vectorize, but not parallelize. All large-scale computation is multi-core.
This still seems like an overstatement, but maybe it depends on what you mean by "most demanding level." I work on systems for the Rubin Observatory, which is going to be the largest astronomical survey by a lot. There's a bunch of C++ certainly, but heaps of Python. For example, catalog simulation (https://www.lsst.org/scientists/simulations/catsim) is pretty much entirely in Python.
Take a look at `lsst/imsim`, for example, from the Dark Energy collaboration at LSST: https://github.com/LSSTDESC/imSim.
Maybe this isn't the "most demanding" but I don't really know why.
> But if A and B are numpy arrays, then A + B will calculate the elementwise sum on a single core only, correct? It will vectorize, but not parallelize.
That's correct, but numba will parallelize the computation for you (https://numba.pydata.org/numba-doc/latest/user/parallel.html). It's pretty common to use numba's parallelization when relevant.
EDIT: Yes, numba is impressive.
That's basically the answer.
However, Numba is quite limited because it only works well for mathematical code (it is not able to apply its optimizations to complex objects, like lists of dictionaries), while on the other side Julia's compiler applies its optimizations to everything.
[1] https://discourse.julialang.org/t/comparing-python-julia-and...
* Julia can do loop fusion when broadcasting, while numpy can't, meaning numpy uses a lot more memory during complex operations. (Numba can handle loop fusion, but it's generally much more restrictive.)
* A lot of code in real applications is glue code in Python, which is slow. I've literally found in some applications that <5% of the time was spent in numpy code, despite that being 90% of the code.
That said, if your code is mostly in numba with no pure python glue code (not just numpy), you probably won't see much of a difference.
If Julia can provide both the REPL/debugging experience of a language like Python or MATLAB with a fast enough JIT to use in production it would be an enormous boon to productivity and robustness.
There are a few limiting factors but I don't think they're absolute.
Then there's Matlab, Mathematica, and they are also pretty good but they're closed source/proprietary, so their ecosystem is mostly limited and driven by commercial interests. Nothing wrong with that intrinsically and they're all widely used but it's one way Julia differentiates itself, by making the language open and making money through services.
But indeed, Julia Computing differentiates itself from something like MATLAB or Mathematica by leveraging a strong open source community on which these products are developed. These products add a lot of the details that are generally lacking in the open source space, such as strong adherents to file formats, regulatory compliance and validation, GUIs, etc. which are required to take such a product from "this guy can use it" to a fully marketable product usable by non-computational scientists. I will elaborate a bit more on this at JuliaCon next week in my talk on the release of JuliaSim.
It also pretty much solved my version of the two-language problem, but that means different things to different people so ymmv.
Such a question seems sort of in bad faith (or loaded), since the selling points of Julia have been hammered time and again on HN and elsewhere, and are prominent on its website. It's a 1 minute search to find them, and if someone is already aware that there's this thing called Julia to the point that they think it's made to be "a big deal", they surely have seen them.
So, what could the answer to the question above be? Some objective number that shows Julia is 25.6% better than Java or Rust or R or whatever?
But first, who said it's a "big deal"? It's just a language that has some development action, seems some adoption, and secured a modest fundng for its company. That's not some earth shattering hype (if you want to see that, try to read about when Java was introduced. Or, to a much lesser degree, Ada, for that matter).
You use a language because you've evaluated it for your needs and agree with the benefits and tradeoffs.
Julia is high level and at the same time very fast for numerical computing allowing you to keep a clean codebase that's not a mix of C, C++, Fortran and your "real" language, while still getting most of the speed and easy parallelization. It also has special focus on support for that, for data science, statistics, and science in general. It's also well designed.
On the other hand, it has slow startup/load times, incomplete documentation, smaller ecosystem, and several smaller usability issues.
But this only concerns when I'm writing the code myself. If I read some code and I see a few nested function calls, there's a combinatorial explosion of possible types that gives me vertigo.
> complex and rational numbers can be constructed using anything that has arithmetic defined.
seriously? this does not seem right, it cannot be like that. If I build a complex number out of complex numbers, I expect a regular complex number, not a "doubly complex" number with complex coefficients, akin to a quaternion. Or do you? There is surely some hidden dirty magic to avoid that case.
help?> Complex
search: Complex complex ComplexF64 ComplexF32 ComplexF16 completecases
Complex{T<:Real} <: Number
Complex number type with real and imaginary part of type T.Thanks for writing this. I think it is an important concept for getting started with Julia. When I tried Julia, I was initially confused and concerned about the subtyping hierarchy, which as far as I understand is undocumented. "Apart from a partial description in prose in Bezanson [2015], the only specification of subtyping is 2,800 lines of heavily optimized, undocumented C code." [0]. Assurance that users can safely ignore the subtyping hierarchy if we maintain semantic equivalence between methods, and that this actually works out in practice, makes it easier to commit to using the language.
For a negative example of this, C++ introduced function overloading — which is a static and therefore broken version of multiple dispatch (that calls the wrong “methods” based only on static type information). They then immediately decided to abuse the bitshift operators for I/O — in the standard library, no less. (There are other abuses of overloading but this is the most famous one.) So that didn’t exactly create a culture where people respect the semantic meaning of overloaded functions. As a result, C++ has given function overloading a really horrible reputation. Partly because it is likely to be not actually so what you want because it’s static rather than based on the actual types of arguments, but even more so because there’s no culture of semantic consistency in C++, starting from the standard library itself — you cannot trust anyone to respect the meaning of anything, not even the language authors.
In Julia, on the other hand, we’ve always been very strict about this: don’t add a method to a function unless it means the same thing. We would never dream of using bitshift operators to do I/O. Since meaning is the level where human thinking operates, this makes it reasonable to write generic code that works the way you meant it to: you can call `+` on two things and just trust that whoever has implemented `+` for those objects didn’t decide to make it append a value to an array or something wonky like that. Not that people haven’t proposed that kind of thing, but because of the culture, it gets shot down in the standard library and elsewhere in the ecosystem.
But yeah, it’s hard to see how this would happen because there is nothing technical preventing the same kinds of abuses that are rampant in C++, it’s just the invisible but extremely force of culture.
Or the sum operator for string concatenation, which is the epitome of non-commutative operation.
I like your point of view, but still I'm skeptical of function and especially operator overloading. Shouldn't these semantic constraints that you mention be enforced by the language? For example, the language does not let you overload + for a non-commutative operation, and so on.
I was surprised to find Julia uses the multiplication method for string concatenation. For me it feels as wrong as using bitshift operators for I/O but maybe I'm missing something.
Multiplication for concatenation makes perfect sense: it is commutative, associative, and consistent with mathematical notation. A space would be even better. And the empty string would be great, but maybe difficult to implement if you want to allow multiple-letter variable names, as julia seems to do.
Even if using multiplication to join strings may be consistent with some properties of mathematical notation, it's less consistent with the usual meaning of the words (I checked the cosine similarity using GloVe 6B words embeddings, the similarity between "add" and "join" is 0.4 while between "multiply" and "join" is only 0.03), so it's probably the balance between being the general purpose and math oriented language.
> For example, the language does not let you overload + for a non-commutative operation
It does allow it? I'm not even sure how one would disallow non-commutative operations. How would you know if a definition is commutative?
> how one would disallow non-commutative operations. How would you know if a definition is commutative?
I don't think that this is possible without solving the halting problem. But in practice, you can do that by documenting this enforcement and making it unfeasible to overload a commutative operator with non-commutative code. For example, the callers of the overloaded commutative operator can and do assume commutativity to optimize compilation; as in, you fill a matrix with "f(i+j)" and it may only evaluate the upper-half of the matrix (this is even more interesting for the associative case). I think that Mathematica does a similar thing, I recall several symbols for operators to be overloaded assuming certain symmetries. As another example, in C++ you must overload "<" with something that is an order relation, otherwise the sort function fails. Being more fancy, a test option of the interpreter may run each call of the operator in a random order, etc.
Also, related to Mathematica, using juxtaposition for product is very natural to mathematicians. It is indeed hard point of friction when moving from Mathematica to Maple or Matlab.
That’s not to say that there are no disadvantages to Julia. I personally see Julia as a beefed up, new and improved R.
However, Julia is perfect if you write mathematical/physical/engineering simulations and data analysis codes, which is my typical use case. Its support for multiple dispatch and custom operators lets you to write very efficient code without sacrificing readability, which is a big plus. Support for HPC computing is very good too.
That's the case now, because Julia made a design decision to focus on extreme composability, dynamism, generic codegen etc which involved compiler tradeoffs...but it's not inherent to the langauge.
For scripting, interpreted Julia is coming. For executables, small binary compilation is as well...particularly bullish on this given the new funding
Citation for this? Julia has had a built-in interpretted since 1.0, in 2017 use `--compile=min`, or `--compile=none` to make use of it. And JuliaInterpretter.jl has been working since 2018. Both are very slow -- slower than adding in the compile time for most applications. As I understand it, this is because a number of things in how the language works are predicated on having a optimizing JIT compiler. As is how the standard library and basically all packages are written.
Julia is going to over time become nicer for scripting, just because of various improvements. In particular, I put more hope on caching native code than on any new interpreter.
That's a bit of an understatement. It's about as fast as C and Rust (ignoring JIT compilation time).
designers seems to have a good amount of PLT knowledge and made good foundations