Can you call Julia routines from other languages? Otherwise you sort of have to go all in...
With PackageCompiler.jl [6] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed.
Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [7] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype)
[1] https://docs.julialang.org/en/v1/manual/embedding/
[2] https://pypi.org/project/juliacall/
[3] https://www.rdocumentation.org/packages/JuliaCall/
[4] https://github.com/Clemapfel/jluna
[5] https://github.com/Taaitaaiger/jlrs
This seems false to me. StaticCompiler.jl [1] puts in their limitations that "GC-tracked allocations and global variables do not work with compile_executable or compile_shlib. This has some interesting consequences, including that all functions within the function you want to compile must either be inlined or return only native types (otherwise Julia would have to allocate a place to put the results, which will fail)." In practice, this means that you can therefore not use the base library in your own programs. PackageCompiler.jl [2] has the same limitations if I'm not mistaken. So then you have to fall back to distributing the Julia "binary" with a full Julia runtime, which is pretty heavy. There are some packages, such as PySR [3], which do this. It seems pretty usable for research I'd say, but difficult to put in production.
There is some word going around though that there is an even better static compiler in the making, but as long as that one is not publicly available I'd say that Julia cannot easily be called from other languages.
[1]: https://github.com/tshort/StaticCompiler.jl
Calling julia from another language does not in general require AOT compilation, though it does help to make things more portable and self contained. If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language.
> PackageCompiler.jl [2] has the same limitations if I'm not mistaken
You are mistaken. PackageCompiler works by basically bundling an entire julia system image with your program and the whole compiler, runtime, and stdlib compiled into one bundle together. If a piece of code works in the REPL it'll work from PackageCompiler
Okay but how do you distribute the program that depends on Julia to clients? You then also need to ensure that they have the right Julia available on their system. That's much heavier than dynamic binaries (which rely only on some shared libraries). My point was that it's not always "easy" to call Julia. That it's easy in HPC is a reasonable statement since you probably have full control over the system and it has loads of RAM available, but I wouldn't call it easy in general.
> You are mistaken.
I guess we're both right. I was talking about the "libraries" functionality [1].
[1]: https://julialang.github.io/PackageCompiler.jl/dev/libs.html
That's not the question that was asked though. The question the person asked as "Can you call Julia routines from other languages?" and the answer to that is "yes, you can easily do that".
__________________________________________________
> I guess we're both right. I was talking about the "libraries" functionality
I'm confused, what functionality exactly are you saying is missing from the libraries functionality? That very example you link to shows one using the julia runtime via printing.
And yes, dynamic dispatch and dynamic code generation do work from libraries created by PackageCompiler.jl as well...
That said, I think the bridges are actually pretty developed compared to what exists between many pairs of high-level languages. They are clearly working for some people since you will find Python-wrapper packages for Julia code. I'm not so sure it's so important to have a single `.so` package. There are solutions which allow Python to be-in-charge and bundle Julia to various extents. There are at least:
1. https://github.com/jlapeyre/julia_project
2. Usage of Conda and https://github.com/JuliaPy/pyjuliapkg
edit: just found https://docs.julialang.org/en/v1/manual/embedding/#Thread-sa...
Ok, so, it's tricky, but at least you can start an interpreter per thread...
It produces quite big binaries, but the more involved and complicated your application is, the less that matters.
The biggest selling point of Julia is that it's as easy to write as python if you don't care about performance, and if you do care about performance it's as fast as C++ if you can just follow some simple principles. This actually works in practice. For my day job I write the slow path exactly as I would python, and that's a really fast way to write code.
If you want a slow language that can call fast code, you already have python.
For me this is the biggest point for giving up on Julia. Julia has a lot going for it in the basic design and I was really excited about it in the beginning.
But even after all these years the environment is sadly almost unusable if you want to do something saner than REPL/notebook.
The TTFP is too damn high and I don't think it will ever be fixed.
I work in "one of those industries" where failure can easily cost millions in the space of minutes, and we use Julia almost exclusively. Unfortunately I can't give you details because of reasons, other than tell you you're very poorly informed.
> you're locked in to Julia
You're not "locked in" to Julia any more you're locked in to Python. Arguably less, as Julia can call C natively, Python can't.
My point isn't that you can't interop, and it seems you're intentionally (?) equivocating on this point. Surely you didn't misinterpret what I wrote as "you can't iterop"? My point is that you can more benefits if you don't. Like I said, if you want a slow language that can call fast routines, why not just use Python.
But I know from experience that this debate will not go anywhere. It's is the same old refusal to see the problem: "you're using it wrong".
Julia is dying and this is why it will die.
I vaguely recall a post from an ex Julia dev who had raised some issues regarding correctness of computations in Julia; I am not very familiar with the situation, have these concerns been addressed?
Edit: now that I recall better, I think the correctness concerns had to do with ambiguous composition of disparate components due to vague interfaces. I think this has to do with multiple dispatch in Julia, but I don't know details. I would love any info about this.
You'd have to be more specific. Currently ambiguous dispatch throws an exception. I don't think this is a recent change either.
You might have a function:
foo(x,y) = x + y
and multiple dispatch means you can call that with integers, or floats, or arrays, or GPU-resident arrays, or automatically-differentiating numbers, or symbolic algebra terms, or...
Just because you can, does that mean you should? It depends...
foo(x::Number) = x+1
And then someone else creates a new type thats a subtype of Number. And they run foo(x) and get errors or unexpected output.
Problem is the new type they created doesn't follow the assumptions that foo expects. Throw multiple dispatch into the mix and it gets even harder.
> Problem is the new type they created doesn't follow the assumptions that foo expects.
If you define that function and someone else (i.e. a user of your lib) defines a subtype of Number and calls your function on it, and it fails, then they haven't respected the interface to Number. There's nothing wrong with that function or type specialization in that case. In that case it's all about defining useful interfaces and respecting, as much of programming is.
E.g. if someone defines a subtype of Number because he has something that's kind of like a number in some respects but not others, maybe that shouldn't be an subtype of Number.
You also shouldn't define functions that have obscure conditions for being properly useable - at least if you expect your code to be useful more broadly. Nothing here is specific to Julia though.
If you have other specific examples we could discuss more.
using Unitful: m; foo(2.0m)
Probably this means you should define `foo(x::Number) = x + oneunit(x)` to respect the Number interface. But this interface isn't very strictly defined. I believe `Base.oneunit` was added after someone started writing the Unitful package -- building something useful in a legal grey area, and formalising later?
I don't know/use Unitful, but if that function didn't fail it's because the guys who wrote Unitful defined a promotion rule from Int to their Unit thingy. So.... that's how their type works. Don't use it. Or better, open an issue on github, they might have an explanation that's escaping us. I suspect it has to do with their mental model of what Unitful is supposed to achieve.
Let me tell you that my intuition agrees with you. As an ex-physicist, if I was designing a lib called Unitful I wouldn't let you sum sum 1 meter plus 1 unitless thing.
EDIT:
Actually, I just tried running your code and I do get DimensionError
foo(2.0m)
ERROR: DimensionError: 2.0 m and 1.0 are not dimensionally compatible.
EDIT2: Sorry I misread you. You do get an error. Ok I see your point. Maybe the Julia docs should more explicit about what the Number interface entails. Is `+ 1` allowed? You're assuming the answer is obvious, but it's not to me. In particular, that's not generic at all. You're probably right about `+ oneunit(T)`
Which country are you in?
In any case, if the complaint is startup time, I do agree that this has been a bit of a thorn in the side. It also interacts poorly with a (in my view) pretty immature ecosystem for running services (for instance, the grpc library doesn't seem to be widely used or heavily maintained) or interacting with it via IPC. So it's great if you can spin up a notebook or repl and leave it running all day doing your work, or if you have a long-running batch process, but I agree that I've found it a bit tricky to interact with it from the outside world in the "usual" ways.
Having said that, it does work - for instance, a system I work on spins up a long-lived julia thread from within a python service using the very nice shared library, and calls into julia functionality through that thread - I've just found it to be tricky; but IMO still very much worth it.
Nowadays also called TTFX where X can refer to variety of things.
In any case, yep, makes sense, thanks!
I guess for the typical scientific experimentation with plotting use case, why not just use the repl / notebook approach that (I think?) doesn't really suffer from this issue, as it's already up and running? My experience with startup time being problematic is when trying to "productionize" things that are not within a scientific / plotting loop. But my impression has been that it works well when doing things that are. Am I missing it?
My scientific/plotting loop runs from Bash (something not solved by Pluto.jl). This allows me to interoperate with a lot of software that's not in Python. I can version the code with e.g. git and the analysis pipeline tends to get refactored into libraries and CLI tools in the process. With Julia this is in practice impossible.
I do wonder if Pluto.jl might solve a lot of these issues for you though. I haven't really used it much, but it seems like it works with just plain .jl files, so you can non-jankily version them (unlike .ipynb), and extract libraries and refactor in all the normal ways.
My frustrations trying to do normal-software-development-principles with julia are actually more about the convention the community has around structuring files and modules, which just hasn't clicked for me at all and is a frequent source of confusion. (The testing framework / layout / something about it also hasn't clicked for me yet either, and I wonder if it just isn't great and that's why people don't seem to write many tests...)
But there's also just so much great stuff about the language that I'm not even really mad about any of this :)
That is more accurate.
> I do wonder if Pluto.jl might solve a lot of these issues for you though. I haven't really used it much, but it seems like it works with just plain .jl files, so you can non-jankily version them (unlike .ipynb), and extract libraries and refactor in all the normal ways.
It may solve enough issues at least for some cases. I'll check it out.
> My frustrations trying to do normal-software-development-principles with julia are actually more about the convention the community has around structuring files and modules, which just hasn't clicked for me at all and is a frequent source of confusion. (The testing framework / layout / something about it also hasn't clicked for me yet either, and I wonder if it just isn't great and that's why people don't seem to write many tests...)
From what I remember from the last time the module system was indeed quite a pain. The using/include/use seems like a total mess, and that IIRC Stefan Karpinski dismissed files-as-modules solely on the reason that Python does it that way did not really bring my hopes up.
I appreciate that the multiple dispatch design (which I find very much a pro for Julia) does bring about some additional challenges for a module system. But my hunch is that majority is because Julia was so strongly made by MATLABists to MATLABists (can replace MATLAB with e.g. R or Mathematica) that basic features required for a general purpose language were not much taken into account.
> But there's also just so much great stuff about the language that I'm not even really mad about any of this :)
This is exactly why I'm so mad about this! I would love to use all that great stuff AND follow good programming principles, but now I can't.
Time To First Plot. It was a meme back then that you needed to wait several minutes to draw just a simple line plot after starting up a Julia REPL/notebook because of compilation times (whereas in Python you're done in seconds with matplotlib.) Seems this has been reduced in Julia 1.9 with automatic pre-compilation of packages, but plotting still isn't "instant" as you would expect in scripting languages like Python.
Everybody doesn't use the REPL. In my, I think quite well founded, opinion nobody should use the REPL for more than one or two lines.
I don't find my plt.plot(x,y); plt.show() in a script ideal either. Something like Pluto.jl (given it's easy to modularize and preferably can be run without a notebook environment) is most likely better for a typical data analysis workflow (and will likely lead to more reproducible results and better tracking of result provenance).
For different cases, e.g. application development, the latency is not that critical. Although I find it quite insane that we accept so slow code iterations this day and age.
If the problem was just running a daemon, it would be an inconvenience at worst. The problem is that changes to the code tend to trigger a recompile, can't be made (e.g. structs can't be redefined if I recall correctly) and most crucially the session can get "silently" into some unexpected confused state.
If you don't need to change the code (and it's structured well enough not to hold a state between calls), then running in a process is no problem. But for that you can do just plain old AOT.
Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. Scientists don't learn to modularize their code and don't learn to understand the basic control flow. I see this all the time with my colleagues and the "pedagogical problem" of notebooks/REPL is very common when I teach programming. E.g. a common problem is that students don't understand variable assignment because with REPL/notebook it doesn't necessarily follow the normal program flow.
I'm kind of amazed how little the scientific programming scene is concerned about this. From software development perspective (I'm an ex software developer turned scientist) having clearly defined state transitions, independent modules of code and the code doing what it says are really really fundamental things, without which you have no chance in hell in making anything more than a few lines to not turn into an unmanageable and buggy mess.
Maybe there's a (mis)conception that scientific code is somehow fundamental different beast than "general programming". But it's not. It's just a relic of having bad systems languages like C and C++ and bad special purpose languages like MATLAB and R. That Python can be and is used for both is an existence proof that there is no such divide. But we need a better Python for both general and scientific programming.
So, two things here: First of all, I have greatly appreciated the discussion, so don't feel like you've over-aired your thoughts!
But also, here's this, in my view, hyperbole again, I just don't see how "more or less forces" is the right way to describe the state of choosing whether or not to use a notebook/REPL style for julia. It works perfectly fine to develop with a different style. There's a little bit of startup time, but it's just not nearly a big enough deal to say it "forces" a different development process.
I think your criticisms of notebook driven programming make sense, but are also slightly overblown; in my experience, scientists are perfectly capable of writing good modular code, despite preferring to work in notebooks most of the time. It's not not a problem, I just don't think it's been a very big problem, in my experience.
But I very much agree with your last two paragraphs. I just don't think notebooks / REPLs are a big part of the reason things have ended up this way. I think it's more of a cultural thing that gets passed down from generation to generation, and it has to do with respect; I think scientists have never taught one another that code deserves respect and professionalism. I think they have always thought of it as "some throwaway stuff that generates the data for the paper"; the paper is the thing deserving of respect, not the tools used to create it.
But I think this has been slowly changing as more and more scientists are digitally native and learn increasingly more software skills as a first class concern.
More accurate is that the REPL/notebook style is currently so much more convenient to get started and to "explore" with that people will use that and it's what all Julia examples etc teach. Pluto.jl may be a good "transition" from this though, although I'm afraid modularization is still too inconvenient with it.
The script-workflow latency is also a reason why "scripters" don't switch to Julia even though they otherwise would.
So while the it's not practically impossible as I hyperboled, it is IMHO a blocker for wider Julia adoption and a transition to better programming practices.
In my experience in both teaching and helping working scientists with their code problems I think the current situation that REPL/notebook is so much easier to get started with is at least a major reason why better practices don't get adopted.
Scientists themselves doing any programming is quite a recent thing. Usually it was "lab engineers" doing "the coding" and data mangling and the scientist doing "the analysis" in something like SPSS or Excel or copypasted R oneliners.
It's also really understandable why scientists don't make much effort to learn and enforce good programming. It's a relatively minor part of their job, they manage to scrape much of the analysis together with poor practices, and when it almost inevitably explodes as the analysis gets more complicated, someone "more technical" gets called to fix the mess (I do this all the time).
In most languages modularizing code is made needlessly hard, and I'm smelling something like this in Julia module system (and in e.g. Python packaging). I sometimes wonder if there's some (unconscious) gatekeeping to keep "the using" and "the programming" (and "the core development") apart artificially. You actually sometimes encounter explicit statements like this (Linus Torvalds famously about C++ but in this post's discussion something like that not knowing R means that you shouldn't use ML (which is extra bizarre because very very few actually know R)).
The benefits of modularization are also not easy to foresee until you learn from many hard lessons of ending up with unmanageable sphagetti or analysis that is wrong but it may be even impossible to pin out why due to lack of reproducible code. And if you don't know about the ways of avoiding/mitigating this, you're not really equipped with learning the lesson.
When this compounds with significant hurdles to do "the right thing" I find it almost inevitable that "the right thing" will not get adopted.
As a sidenote I find it also quite odd why so little effort is done to "show the work" with data analysis when it's literally required when you do it in math class. I think they are almost the same thing. Maybe it stems from the computer seen as a calculator and "the work" is done on paper, but this is mostly due cultural lag from the era where computers were in practice calculators.
TTFP is (close to) solved. 1.9.0 TTFP is under 3 seconds for me, which I find about acceptable for script-type workflow. However, time-to-first-DynamicalSystems is still around 10 seconds, which is not.
I just installed and ran on my macbook - took me 7 minutes to get it installed and run through to a first plot, most of that was installing and compiling the Plots library.
Once it's installed and available I can run it in about 1 second and then execute
using Plots
function test_plot()
x = range(0, 10, length=100)
y = sin.(x)
plot(x, y)
end
@time test_plot()
which gives me a result of 0.048700s and a nice plot.
I don't think this is the defining issue with Julia now - it definitely used to be 10 years ago - but now it's more than competitive with Python in this regard.
The issue is really support and buy in for all the new cool libraries that folks make available to use.
$ time ./julia-1.9.4/bin/julia ttfp.jl
0.325806 seconds (202.80 k allocations: 13.208 MiB, 84.23% compilation time)
real 0m2.401s user 0m2.357s sys 0m0.553s
I.e. the TTFP is 2.4 seconds, not 0.33 seconds.
That said, 2.4s is huge progress (e.g. with Julia 1.6.0 it's 10 seconds for me) and acceptable for most simple plotting.
But just adding e.g. "using DynamicalSystems" brings this to 14 seconds, which is not acceptable for iterative editing. And it makes any Julia program using "DynamicalSystems" wildly impractical to use in a pipeline with other programs.
$ time ./julia-1.10.0-rc2/bin/julia -e "using Plots; using DynamicalSystems;"
0m5.433s
This is a different conversation from
julia> @time plot(x, y)
0.000465 seconds (484 allocations: 45.992 KiB)
Wouldn't something like slow interpreter option for "glue code" be feasible? Most code doesn't need to be fast, and waiting for the "hot functions" to compile on change wouldn't be too bad. And you could still AOT for maximum speed when needed. This is essentially how Javascript JITs manage to start up so fast.
Another fine "hack" for many cases would be to keep the Julia process running (like a daemon) but making sure the state is clean when a script is run on it.
Something like PackageCompiler is workable, at least for the short term. Long compiles are OK if they don't have to be done all the time. I indeed manage to get < 1s TTFP with DynamicalSystems with it (last time I tried a while ago I didn't manage). The UX for creating sysimages could be nicer, but doesn't seem anything that a quick script can't fix. Maybe I'll give Julia another go in my next analysis.
From what I gather there's now a new approach for caching the compilation results. I really hope it will succeed, but I'm a bit jaded from the many "Julia 1.x fixes the TTFX" news.
The script typically loads the data files and runs simulations (a lot of times when their parameters are optimized). This means the models have to be fast, but they are also recurrent in nature so they can't be vectorized to numpy. So depending on the case it's usually numba, cython or C++, all of which can be quite painful.
Can't you compile the pipeline into a single script and then run it in a single instance of whatever?
Or at least semantics of that. The steps should be independently callable and each call should result in the same output with same parameters, i.e. the state would reset for each run. I don't care so much how this is exactly implemented as long as it fullfills these.
The data doesn't need to stay in the process memory. It's can be e.g. mmapped and is cached by the OS anyway. And deserializing tens or even hundreds of megabytes takes usually less time than Julia takes to import Plots.
This is the thing that isn't feasible with Julia due to the TTFP (each step incurs the startup latency).
> Wouldn't something like slow interpreter option for "glue code" be feasible? Most code doesn't need to be fast, and waiting for the "hot functions" to compile on change wouldn't be too bad. And you could still AOT for maximum speed when needed. This is essentially how Javascript JITs manage to start up so fast.
That is something I've been meaning to investigate more. A lot of the Plots.jl code is internally dynamic because of its design to hold everything in a Dict{Any} and recurse the plot recipes. In other words, it doesn't even benefit from having the JIT at all because internally everything is uninferred boxed variables. There's probably a good way to just do "please interpret the calls in this part of the pipeline" and not see any runtime difference but chop out the core of the compilation. It's relatively straightforward to do, it's a one liner using JuliaInterpreter.jl @interpret, and I think one dig that results in a win would be a nice example of what should be a pattern we do more often in the near future.
For now the interpreter seems to focus on debugging, and if I understand correctly, it can't be really used to e.g. speed up imports or to specify hot code to be AOT'd/JITed?
Do you have anything that you've shared publicly?
Codewise it's not pretty, especially the data mangling, but other people have managed to use it for their own purposes.
The model had to be implemented in C++ and wrapped for python bindings. It's a pain but sadly less painful than doing it with Julia's startup time.
Ok thanks for sharing.
(I still disagree about everything you said about Julia though. I bet you I could re-implement this faster in Julia thank your C++, and in a type generic way too which would make it trivial for users to plug in their types, and it would be trivial to package - I'm assuming you didn't package that repo, you just expect your users to be shuffling files around. You're not contributing to improving the reproducibility situation in the academia, let me tell you xD)
I didn't package it and the data mangling part has very little use outside this specific experimental design. Also Python's packaging is quite shitty.
The C++ code is totally standalone and can be git-pulled and compiled how you like. C++ packaging is totally abysmal.
As I said in the comment and the README the code is a mess (although a piece of art compared to the notebook shit out there). It's purpose is that the results can be reproduced from the raw data.
I would love to make my analysis code cleaner. But there's zero incentive for that and very little time. Nobody sadly cares about the code, and most scientists can't code for shit. This is probably why Julia was designed so that it forces your code to be shit.
Sorry but the tone, but you kinda set it.
JS does this by multi-phase JIT. Python by being dog slow in the execution and AOT by being dog slow in the compilation.
I think Julia's "JIT-AOT" may be a good approach (and perhaps almost necessary for Julia's dispatch) too but it seems to be very hard to make start up fast.
edit: oops, maybe PythonPlot is what i should use, not PyPlot. But when I try that I see: ┌ Warning: `PythonPlot` 1.0.3 is not compatible with this version of `Plots`. The declared compatibility is 1 - 1.0.2.[ and doesn't seem to be any faster