Why We Use Julia, 10 Years Later
julialang.org
julialang.org
- But the REPL lacks the ability to redefine structs on the go (which I can understand as it'd be tough to do, or simply not possible). But that, combined with the slow start up time makes life a bit harder than it should. Fortunately, one doesn't redefine its structs every day.
- There are also lots of libraries but the quality of the documentation is often sub par. For a newcomer like me, working examples would be great. For example, if you use the plots library, you'll have hard time finding a list of all possible plots (the documentation talks about lots of things, but strangely, not a list of possible charts). I've also looked at doing linear regression and GLM and, again, you've code libraries but the examples are rare...
- the startup time are still quite slow but that's ok because somehow you adapt your workflow around.
- being able to use greek symbols as identifiers is super cool but your editor as to manage that, else you'll have to memorize shortcuts...
But still, I keep on using it, it's much faster for my use case (data processing). I mean, faster than r or Python (for which I could write fast code but that'd mean I'd have to change the way it is written)
It is possible to redefine structs in Pluto.jl which is also a productivity booster overall due to its reactivity.
> For example, if you use the plots library, you'll have hard time finding a list of all possible plots (the documentation talks about lots of things, but strangely, not a list of possible charts).
The Makie.jl plotting library has really great docs nowadays: https://makie.juliaplots.org/stable/examples/plotting_functi...
Even then it lacks geographical plots, and the GeoMakie.jl (part of the same ecosystem) documentation is limited.
https://juliaplots.org/GeoMakie.jl/stable/
I know, I know, I should contribute documentation.
ProtoStructs.jl: https://github.com/BeastyBlacksmith/ProtoStructs.jl
You might benefit from numba. I've used it to speed my Python up enormously and completely painlessly just by adding decorators to critical functions. It's why I'm not considering moving to Julia.
YMMV, but for me I found that there were many unsupported parts of Numpy I had to work around which meant I was effectively doing a full rewrite.
Especially assignments using boolean masks and working with multi-dimensional arrays in general is really tough
The beauty of Julia is that relatively naive Ruby-like code is already quite quick. And if you implement inner loops in an imperative way, with an eye towards not generating excessive allocations, it can approach C++ speed while still being nice high-level code that is close to mathematics or business logic.
Besides, the other strong point of Julia is composability. The ecosystem is made up by lots of small libraries that can interact in ways the original designers did not expect or plan for. In contrast, Python has exceptional libraries, but they tend to be big monoliths.
The problem with Julia right now is that some libraries are not sufficiently mature. For example, there's no mature native replacement for XLA or PyTorch. I know about Flux, but it's nowhere close if you wanna create, say, a large transformer. Or say you are working with GLMs. GLM.jl is nowhere close to R.
Some other Julia libraries represent the state of the art, though. I just can't wait to get all foundations complete! It's a really promising space for probabilistic and differentiable programming.
Python still doesn't feel like a natural fix for data science work. I am guessing it is more bias opinion but why based on 0 for this domain????
But it means that you have to work with arrays. For me it often means breaking the flow of my (code) explanation to migrate to other data structures. Sure it is then fast, but it gets less readable and harder to update.
Now, I'm a programmer at heart, so I think in the "functional" paradigm, not the array/signal one. There's some "impedance" I guess :-)
Or to put it another way-- if I'm using vectorized code in Numpy I can't deal with the external ecosystem from within my vectorized code either.
> The code is fast, but you lose the ability to organize your data the way you would in regular python,
Can you clarify what you mean by this? I lose the ability to organize my data the way I'd like if I'm vectorizing my code too.
Also, the comment you replied to was explicitly comparing Numba to vectorized Python, so you should not abandon that comparison in your reply without saying so.
The comment you replied to explicitly says "regular python".
I have tried to write an application targeting HPC and in itself is not very complicated (probably ~2000 lines or in that order.) But I did things like using the Python language as the metaprogramming language for Numba (basically higher-order function where you jit inside.)
All in all my experience of Numba tells me that if I am designing the same package now I'd write it in Julia where jit is "first class" and you don't need to constantly thing about the boundary between Numba and Python.
One thing I think R does not get enough credit for is really strong enforcement of documentation. If you want to get a package in CRAN it is going to be more work to not document your library than to just document it correctly. As a result nearly every R package has very solid documentation, including a well formatted pdf manual, a series of "Vignettes" that show how to use the library, and excellent in-interpreter documentation.
- parameters documentation can be understood only if you actually know the theory behind what the library documents. One may say that it's a good thing in the sense it prevents one to shoot himself in the foot but for discoverability, that's painful. I've done basic stuff such as GLM, LDA, PCA,... For example, there are several ways to do PCA's but which one do you choose ? Not everybody know the theory behind each PCA's formulation...
- examples are usually very limited and don't show what the library can do. For example, plots (the one from the base library) are really not well documented. Examples are really scarce and don't even show the graphics themselves.
- The package federation (not exactly documentation) is really bad : there are lots of library that overlap, that redefines symbols here and there without telling, etc.
So R documentation is so so... It feels like it suffers from lots of math explanation I've seen : a strong will to write the minimum possible which makes everything hard to get into.
I like R but documentation is not a strong point. Ecosystem is.
It's a language meant for a specific domain. Expecting domain knowledge in said domain is not a failing - it's logical.
Just hit this one: after rewriting a function I don't see any change in behavior. Reason: I overwrote the function but the more specialized one was being called.
But there are plenty more. After some time it's just better to nuke the REPL and start clean. That said, I love programming in the REPL.
Yeah, I've had this and the struct redefinition problem since the very early days of Julia, that's why I never fully bought into the Revise.jl based development model (it has its good parts, but these are big limitations that should be mentioned more often when recommending it). That's also why I resisted the removal of the `workspace()`-clearing function (like MATLAB's `clear`), since that would be an alternate option for quick and dirty exploration in a lot of cases; though these days the latency problems of exiting to shell and coming back are much less, so it's not as much of an issue.
The normal thing today is comparing Julia to R, Matlab and Python. But my intro to Julia was actually trying to convert code examples in the O'Reilly book: "Exploring Everyday Things with R and Ruby: Learning About Everyday Things" to Julia.
I thought that worked quite nicely and that the Julia code looked a lot nicer than the Ruby code. It made me write a follow up blog post a few days later: https://assoc.tumblr.com/post/71454527084/cool-things-you-ca...
Unlike your typical Julia user I was always into programming because I wanted to make computer games. I liked playing with Game engine architectures and those exposed me to the problem of handling collision between objects of different types. Multiple-dispatch solves that problem very elegantly. Traditional object-oriented languages are pretty bad at it.
From the perspective of writing clean and expressive code, this really won me over to Julia early on before I had done much serious work with it.
It became my goto language for writing Unix tools, replacing Go, Ruby and Python.
I'll just have packages for doing doing image file conversions in batch, modifying source code, changing configuration files etc.
Actually come to think about it, I used to have shell scripts in a pipeline to build an application for security. We had to obfuscate the source code and stuff like that. When I rewrote to Julia it all ran much faster. Even if there was a startup time, the workload was heavy enough that the higher performance of Julia easily outperformed bash.
High level functional style code with things like map and filter can frequently be JIT compiled to optimal machine code.
Fortran is considered faster for numerical code than C and well polished Fortran libraries like BLAS is already getting outperformed by Julia.
For typical systems programming with need to tight control of memory and real time system C will still have the edge. But for anything crunching lots of numbers like data analysis or machine learning Julia will likely outperform everybody else.
Currently there can some nontrivial compilation latency with this approach, but since LV ultimately emits custom LLVM it's actually perfectly compatible with StaticCompiler.jl [2] following Mason's rewrite, so stay tuned on that front.
Link here for others: https://octavian.julialinearalgebra.org/stable/
> The normal thing today is comparing Julia to R, Matlab and Python.
I'm on the same page as you, though. I've never written R or any Matlab, but when Julia was still new, I remember seeing an example that it was a language that didn't use S-expressions, yet had as potent of macro and meta-programming abilities as if it did. Seemed like it'd be wonderful to work in.
Most of the time multiple dispatch and a what really help abstracting your code and reuse.
It's not an overstatement to say that this language totally changed my relationship with computers and programming.
Julia is getting usable even in "normal" applications not only academic stuff, as person who come back after 1.5/2 years to julia i feel like i can use it again in my job cause it is a lot more stable at have a lot of new neat futures + CUDA.jl is amazing.
I hope Julia team will still explore a bit more static type inference and full AOT compilation if language got full support for AOT it'll be a perfect deal for me :).
Here's the feature roadmap https://github.com/tshort/StaticCompiler.jl/issues/59 that should help people understand what currently works and what I think I can reasonably accomplish eventually
Also, the way dynamic dispatch + specialization work is beautiful.
The only change I would recommend is to have a way to distinguish between creating and setting a variable. Without this distinction it is very easy (especially for someone with dyslexia) to misspell a variable and accidentally create a new one instead of assigning a value. Something like this would be nice:
var x = 5
or even something like Go:
x := 5
It may sound dumb, but such a "simple" change (simple conceptually, can't speak of the dev challenge involved) change would be an enormous quality of life increase to me. I've ended up looking into scoping rules and what not when bug-hunting because of this, but often just ended up going back to a language with stricter syntax around variable creation (goes for moving away from Python etc as well). The more the code, the bigger of an issue this becomes.
> Would you be willing to provide an example where this becomes a problem?
> this should be a problem for structured applications and hints at a more fundamental design problem
I absolutely do not trust myself. :)
When things go wrong for this reason (difficult to spot typo etc) I may waste time on investigating whether blocks are actually leaky or not etc - I have issues with Python syntax for the same reason. Going in the other direction, I like Rust's perhaps overly verbose "let mut" much better, since this has other consequences for explicitness (mutable references '&mut X' vs immutable references '&X'). No special tooling or IDE required to spot this, I can 'cat' the code in a terminal and it's still obvious.
I absolutely don't think Julia or Python should "go Rust", since their dynamic nature is a strength, especially so in academia where notebooks are popular. But a simple "I'm creating a variable" should IMO be a nice visual cue to any developer.
> It’s just an interesting thought experiment — we can’t practically do this even in a 2.0 release. It would be a massively breaking change: we would need to disallow implicit declaration of locals by assignment, which breaks all Julia code everywhere. I also think that what we do now is good and less annoying for new users since they will be accustomed to just assigning to initialize and declare new locals if they are coming from Python, R, Matlab or Ruby (to mention just a few).
(What isn't mentioned explicitly is that this would also break a lot of macro code which accepts assignment-like syntax, which would have to deal with this new type of node in the AST and make decisions about how to handle it.)
The good news is that, as Stefan mentions in a thread linked from there, you can already do `local mylongvarname = 5` as an explicit variable creation syntax; and Julia has good enough tooling to parse it (and is getting even better ones as we speak) that you can disallow assignment to any variable that you haven't seen before with a `local` declaration, with a linter-like step. So the linter will error on a later `mylngvarname = 42`.
It's not the language itself adding it as a feature - so code inherited from elsewhere likely won't have it, and you might have to get buy-in from your team - but it's probably as close as you can get to what you want.
[1] https://discourse.julialang.org/t/why-no-operator-for-initia...
It has made me objectively better at my job, and a much better programmer overall.
I would guess there are some possibilities that could be a menace:
1. Someone manages to convince scientists to use a statically typed language.
2. Computing paradigm changes drastically. Everyone moves to quantum computing or similar.
3. In another 10 years someone learns from the lessons of Julia and improves on them to make a much better language.
As a replacement for Python, I've never understood the appeal, and it's probably not going to fill that niche. Still, as a replacement for Matlab/Mathematica, it's doing swimmingly.
All the best.
In that case, I guess we'll see. Python will never be a particularly good language for implementing such things, but has become pretty ok for using them once someone has implemented (in another language, and made the python bindings).
Network effect is the real barrier, python itself is very easy to toss for this use .
Will new projects that need a flexible performant language, that would have otherwise been shoehorned into Python as the closest available option, choose Julia instead? Quite likely, and that happens a lot.
Python has its place, and is good at a lot of things. But there have always been a lot of projects that have straddled the line between numeric computing and general purpose computing, needing both, where Python was chosen for lack of choice, just because it was the best among the bad options. That's the pain point Julia addresses, and the people who have experience being stuck in that situation understand what its place is and why it has the success it has. (And being a well designed language that's a pleasure to use helps too.)
# This is all StaticCompiler-friendly
using StaticTools
function print_args(argc::Int, argv::Ptr{Ptr{UInt8}})
# c"..." lets you construct statically-sized, stack allocated `StaticString`s
# We also have m"..." and MallocString if you want the same thing but on the heap
printf(c"Argument count is %d:\n", argc)
for i=1:argc
# iᵗʰ input argument string
pᵢ = unsafe_load(argv, i) # Get pointer
strᵢ = MallocString(pᵢ) # Can wrap to get high-level interface
println(strᵢ)
# No need to `free` since we didn't allocate this memory
end
println(c"That was fun, see you next time!")
return 0
end
# Compile executable
using StaticCompiler # `] add https://github.com/tshort/StaticCompiler.jl` to get latest master
filepath = compile_executable(print_args, (Int64, Ptr{Ptr{UInt8}}), "./")
shell> ./print_args 1 2 3 4 5.0 foo
Argument count is 7:
./print_args
1
2
3
4
5.0
foo
That was fun, see you next time!
shell> hyperfine './print_args hello there'
Benchmark 1: ./print_args hello there
Time (mean ± σ): 2.2 ms ± 0.5 ms [User: 0.8 ms, System: 0.0 ms]
Range (min … max): 1.5 ms … 5.5 ms 564 runs
Warning: Command took less than 5 ms to complete. Results might be inaccurate.
shell> ls -lh print_args
-rwxr-xr-x 1 user staff 8.5K Feb 10 02:36 print_args
[1] https://github.com/tshort/StaticCompiler.jl
Installing on windows (workday I'm consigned to flames of woe) is simply a matter of downloading portable install .zip file, a simple startup .bat script[0], 'add Pluto', 'import Pluto' and finally 'Pluto.run()'
I now have an Observable-like[1]notebook with all the Julialang goodness without running an installer
[0] my julia.bat looks like:
@echo off
@REM see https://docs.julialang.org/en/v1/manual/environment-variables/
@REM %%1 is the project directory
set JULIA_PROJECT=%1
set JULIA_DEPOT_PATH=%~dp0JULIA_DEPOT
set JULIA_BINDIR=%~dp0julia-1.7.0\bin
set PATH=%JULIA_BINDIR%;%PATH%
%JULIA_BINDIR%\julia
And I don't think making a web framework would be a focus of Julia since it was built with science and computing mindset.
This might be cliché but use right tools for the job. I still use Ruby for web development and Julia for the science and machine learning stuffs (gradually moved from Python ecosystem).
one thing people don't see is how "free" things are in Julia... we don't need half a million line CMake and C++ then bind to Python to make something work, and we don't need monorepos that re-invent same interface (array, numpy, autodiff etc.) three times. It just works™, and small libraries constantly build on top of something with only ~100 lines and gives you something fast and flexible.
But participating in that "hype" is not necessarily what will entrench julia for the long term. Turning its unique characteristics (unique versus these other two open source contestants, not across the entire programming language landscape) into unmissable developer / user experiences seems to me a safer route. E.g what makes R impossible to ignore is the richness of its statistical toolkit. What makes python impossible to ignore is the productivity boost for typical tasks etc.
> There are so many places in math that are simply untouched because they sound like the domain of a compilers instead of "computational science", and I'm excited to see how the next 10 years bridges this gap in Julia.
https://www.julialang.org/blog/2022/02/10years/#chris_rackau...
I just read this book. Every page was like, 'wow'.
A couple of questions I had afterwards:
* I saw the DateTime type, but it doesn't seem to have a timezone - how do you deal with timezones?
* How is it to work with async code?
* It's a garbage collected language, what are some techniques to avoid performance pitfalls due to that?
If there's other Julia programmers around, any answers would be much appreciated!
For async code Julia has "tasks" which are similar to goroutines in Go: they can be multiplexed on a number of OS threads for parallelism, and channels can be used to communicate with other tasks. The language provides some sugar to make some common things easier, for example to get the result of a long-running operation:
# Make channel and pass it to anonymous function
result = Channel() do chan
# Anonymous function body
sleep(5)
put!(chan, "Done")
end
# Do stuff
...
# Now get the result
take!(result)
Thanks for the chuckle :-)
high energy physics
lots of computational bio
fluid dynamics
rendering
In general, python and matlab really struggle in problems where you want maximum performance, but the most efficient algorithms aren't vectorized. In some cases, this is solvable by writing python libraries in C/C++, but especially in scientific fields, the end users are often the same people writing the algorithms, so they don't gain much from a python library if they have to do all the hard work in C++ anyway. Julia gives them a way more productive dev experience while still having a good user experience.
indeed, the grants (in the millions $) given to rewrite C++ or Python just to handle array (because for loop sucks) and ends up making monorepo blobs is jaw dropping -- while it's almost free in Julia, with 1/100 of the line of code and more flexibility we can match and beat even C++ code...
You're the first person I've ever seen call matplotlib "user friendly".
Anyway, the answer to your question is very simple: Because it's difficult, and a lot of work. Not because they can't be bothered, as your question seems to imply.
* TTFP - this is the big one. Every thing is a sluggish sometimes * Sluggish IDE in VSCode; I am on Windows so start up is a bit of a pain i often have to wait 20+ s for it to become responsive * Non-robust Basic Data Science coverage e.g. GLM fails on some datasets that R can handle. So it's the only code in my workflow that still has R in it
You cannot do `x.foo(y, z)` instead of `foo(x, y, z)`, a feature I sorely miss in Julia. Nim is awesome in this regard. There are some functions / paradigms that are just better represented by `x.foo(y, z)`. Chaining functions in particular in typescript / javascript / rust is SOOO nice. Specifically, in Julia a lot of times I want to create a immutable struct with multiple fields, many of which have defaults. In Julia none of ways seem clean to accomplish this. In rust, something like this is very common:
let m = App::new("My Program")
.author("Me, me@mail.com")
.version("1.0.2")
.about("Explains in brief what the program does")
.arg(
Arg::new("in_file").index(1)
)
m = App("My program")
m = author(m, "Me, me@mail.com")
m = version(m, "1.0.2")
m = about(m, "Explains in brief what the program does")
m = arg(m, index(Arg("in_file"), 1))
With `Base.@kwdef` (or the enhanced `@with_kw` from Parameters.jl) on the struct definition, that example can be:
m = App(name = "My program",
author = "Me, me@mail.com",
version = "1.0.2",
about = "Explains in brief what the program does",
arg = index(Arg("in_file"), 1)
Also, there's Chain.jl for a better pipe operator, though I agree improvements to the Base pipe would be great.
To the original point, the problem with x.foo(y, z) is that it would be highly misleading in that it seems to suggest that x is somehow special to the dispatch of the call, which isn't true. While superficially it can feel more familiar, it would instead become a trap for people giving them a wrong mental model of the call's semantics.
The reason to use immutable structs is better performance when you use them, not when you construct them - how often are you constructing new struct objects that the constructor performance is a significant issue? And if you are constructing many many thousands of objects to the level it becomes a performance concern, any difference from keyword arguments will be easily dwarfed by the cost of actually constructing the struct in memory.
Also, these constructors ar simply passing on these named arguments to the non-keyword constructors. Anything complex you're doing during construction will be happening in the non-keyword constructors, and those do get specialized to have good performance; these keyword constructors are simply providing a small, simple interface to them, in a way that shouldn't affect performance in any measurable way.
It takes a bit time to get into because it merges the concept of packages and environments.
Could you explain what you mean here? The point of environments is to manage packages and dependencies, so by definition they're going to be closely coupled. It seems something about it seemed unintuitive to you though, so I'd like to understand what it was.
Also, Pluto is a marvel - seems to fix all of my complaints about Jupyter and is trivially easy to use.