Any thoughts from someone more plugged in to the community today?
It does have some well-known issues (like slow startup/compilation time) but if you're using it for long-running data pipelines it's great.
If you can't talk about library stacks, it'd be at least interesting to hear your thoughts about how you minimize memory allocation.
We control memory allocation the boring, manual way – we preallocate all our arrays, and then just modify them, so that we have very little continued allocation in production.
As for the actual numerical stuff I tend to roll my own implementations of most algorithms to better control relevant tradeoffs. There are sometimes issues where a particular algorithm is implemented by a Julia package, but has performance issues / bugs in edge cases. For example, in my testing I wasn't able to get ImageContrastAdjustment CLAHE to run very fast and it had an issue where it throws an exception with an image of all zeros. You also can't easily call the OpenCV version as CLAHE is implemented in OpenCV using an object which doesn't have a binding available in Julia. After not getting anywhere within the ecosystem I just wrote my own optimized CLAHE implementation in Julia which I'm very happy with, this is truly where Julia shines. It's worth noting however that there are many excellent packages to build on such as InterprocessCommunication, ResumableFunctions, StaticArrays, ThreadPinning, Makie, and more. If you don't mind filling in some gaps here and there its completely serviceable.
As for the core language and runtime we are deploying a Julia service to production next release and haven't had any stability/GC/runtime issues after a fairly extensive testing period. All of the Python code we replaced led to a ~40% speedup while improvements to numerical precision led to measurably improved predictions. Development with Revise takes some getting used to but once you get familiar with it you will miss it in other languages. All in all it feels like the language is in a good place currently and is only getting better. I'd like to eventually contribute back to help with some of the ecosystem gaps that impacted me.
The other day that old article "Why I no longer recommend Julia" got passed around. On the very same day I encountered my own bug in the Julia ecosystem, in JuliaFormatter, that silently poisoned my results. I went to the GitHub issues and someone else encountered it on the same day. I'm sure they will fix it (they haven't yet, JuliaFormatter at this very moment is a subtle codebase-destroyer) but as a newcomer to the ecosystem I am not prepared to understand which bog standard packages can be trusted and which cannot. As an experiment I switched to R and the language is absolute filth compared to Julia, but I haven't seen anyone complain about bugs (the opposite, in fact) and the packages install fast without needing to ship prebuilt sysimages like I do in Julia. Those are the only two good things about R but they're really important.
I think Julia will get there once they have more time in the oven for everything to stabilize and become battle hardened, and then Julia will be a force to be reckoned with. An actually good language for analysis! Amazing!
The codebase destruction warning was not super loud, though. Obviously I missed it despite using JuliaFormatter constantly. It doesn't get printed when you install the package nor when you use it. It's not on the docs webpage for JuliaFormatter. 2.x is still the version you get when you install JuliaFormatter without specifying a version. The disclaimer is only in the GitHub readme, and I was reading the docs. What other packages have disclaimers that I'm not seeing because I'm "only" reading the user documentation and not the GitHub developer readme?
I don't think this is an accurate summary. the bug here is that JuliaFormatter should put a <=1.9 compatibility bound in its Project.toml if it isn't correct with JuliaSyntax.jl
OffsetArrays was different because it exposed a bunch of buggy and common code patterns that relied on (incorrect) assumptions about the array interface.
Python started in 1989, it also took its time.
Are you using Julia?
it can be used for deep learning but you probably shouldn't, currently, except as a small piece of a large problem where you want Julia for other reasons (e.g. scientific machine learning). They do keep improving this and it will probably be great eventually.
i don't know what the experience is like using it for traditional data science tasks. the plotting libraries are actually pretty nicely designed and no longer have horrible compilation delays.
people who like type systems tend to dislike Julia's type system.
they still have the problem of important packages being maintained by PhD students who graduate and disappear.
as a language it promises a lot and mostly delivers, but those compromises where it can't deliver can be really frustrating. this also produces a social dynamic of disillusioned former true believers.
This is true. As far as I understand it, there is not a type theory basis for Julia's design (type theory seems to have little to say about subtyping type lattices). Relatedly, another comment mentioned that Julia needs sum types.
- simply typed lambda calculus
- System F
- dependent type theory (MLTT)
- linear types
- row types
- and so on
"On the use of LISP in implementing denotational semantics"
https://dl.acm.org/doi/10.1145/319838.319866
Type theory in CS isn't a synonymous with whatever Haskell happens to do.
- Plots.jl, 1.4 seconds (include package loading)
- CairoMakie.jl, 4 seconds (including package loading)
julia> @time @eval (using Plots; display(plot(rand(3))))
1.477268 seconds (1.40 M allocations: 89.648 MiB, 2.70% gc time, 7.16% compilation time: 5% of which was recompilation)Worse, there are still way too many compilation traps. Splatted a large collection into a function? Compiler chokes. Your code accidentally moves a value from the value to the type domain? You end up with millions of new types, compiler chokes. Accidentally pirate a method? Huge latency. Chose to write type unstable code? Invalidations tank your latency.
It's the most annoying thing about hn that people will regularly declare/proclaim some thing like this as if it's a Nobel prize winning discovery when it's actually just some incremental improvement. I have no idea how this works in these people's lives - aren't we all SWEs where the specifics actually matter. My hypothesis is these people are just really bad SWEs.
The problem comes from Julia trying to be two languages at once -- the dynamic language that is useful for quickly generating plots and prototyping; and the production language that's running production code on the backend server, or running the HPC simulation on the supercomputer. They've deliberately staked out the middle point here, which comes with the benefit of speed but the tradeoff is in the ttfp latency. It might be considered the leak of the multiple dispatch abstraction. Yes it can feel like magic when it works, but when it doesn't it manifests as spikes in latency and an explosion of complexity.
In the end I don't know how big the ttfp issue is for Julia. But they've certainly branded it and the existence of the problem has made its way to people who don't use the language, which is an issue for community growth. They've also left themselves open for a language to come on that's "Julia but without ttfp issues".
All I can say is that many of "us" live in that tension between high level and low level every day. It's actually going to become more pronounced with `--trim` and the efforts on static compilation in the near term. The fact that Julia can span both is why I'm a part of it.
$ time julia -e "exit"
real 0m0.156s
user 0m0.096s
sys 0m0.100s
$ time julia -e "using Plots"
real 0m1.219s
user 0m0.981s
sys 0m0.408s
$ time julia -e "using Plots; display(plot(rand(10)))"
real 0m1.581s
user 0m1.160s
sys 0m0.400s