Comment by eigenspace | Hacker News Reader

Comment by eigenspace | Hacker News Reader

Parent Full thread

Yes, julia can be called from other languages rather easily, Julia functions can be exposed and called with a C-like ABI [1], and then there's also various dedicated packages for languages like Python[2], R[3], C++[4], Rust[5] and more to call Julia code.

With PackageCompiler.jl [6] you can even make AOT compiled standalone binaries, though these are rather large. They've shrunk a fair amount in recent releases, but they're still a lot of low hanging fruit to make the compiled binaries smaller, and some manual work you can do like removing LLVM and filtering stdlibs when they're not needed.

Work is also happening on a more stable / mature system that acts like StaticCompiler.jl [7] except provided by the base language and people who are more experienced in the compiler (i.e. not a janky prototype)

[1] https://docs.julialang.org/en/v1/manual/embedding/

[2] https://pypi.org/project/juliacall/

[3] https://www.rdocumentation.org/packages/JuliaCall/

[4] https://github.com/Clemapfel/jluna

[5] https://github.com/Taaitaaiger/jlrs

[6] https://github.com/JuliaLang/PackageCompiler.jl

[7] https://github.com/tshort/StaticCompiler.jl

> Yes, julia can be called from other languages rather easily

This seems false to me. StaticCompiler.jl [1] puts in their limitations that "GC-tracked allocations and global variables do not work with compile_executable or compile_shlib. This has some interesting consequences, including that all functions within the function you want to compile must either be inlined or return only native types (otherwise Julia would have to allocate a place to put the results, which will fail)." In practice, this means that you can therefore not use the base library in your own programs. PackageCompiler.jl [2] has the same limitations if I'm not mistaken. So then you have to fall back to distributing the Julia "binary" with a full Julia runtime, which is pretty heavy. There are some packages, such as PySR [3], which do this. It seems pretty usable for research I'd say, but difficult to put in production.

There is some word going around though that there is an even better static compiler in the making, but as long as that one is not publicly available I'd say that Julia cannot easily be called from other languages.

[1]: https://github.com/tshort/StaticCompiler.jl

[2]: https://github.com/JuliaLang/PackageCompiler.jl

[3]: https://github.com/MilesCranmer/PySR

I think you misunderstood me. Nothing about calling julia from other languages requires StaticCompiler.jl, that's just an experimental proof of concept for small, standalone binary generation with some heavy limitations.

Calling julia from another language does not in general require AOT compilation, though it does help to make things more portable and self contained. If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language.

> PackageCompiler.jl [2] has the same limitations if I'm not mistaken

You are mistaken. PackageCompiler works by basically bundling an entire julia system image with your program and the whole compiler, runtime, and stdlib compiled into one bundle together. If a piece of code works in the REPL it'll work from PackageCompiler

> If one wants, they can literally just spawn a julia process, define a function on the fly and then hook into that process and call that function from another language.

Okay but how do you distribute the program that depends on Julia to clients? You then also need to ensure that they have the right Julia available on their system. That's much heavier than dynamic binaries (which rely only on some shared libraries). My point was that it's not always "easy" to call Julia. That it's easy in HPC is a reasonable statement since you probably have full control over the system and it has loads of RAM available, but I wouldn't call it easy in general.

> You are mistaken.

I guess we're both right. I was talking about the "libraries" functionality [1].

[1]: https://julialang.github.io/PackageCompiler.jl/dev/libs.html

> Okay but how do you distribute the program that depends on Julia to clients?

That's not the question that was asked though. The question the person asked as "Can you call Julia routines from other languages?" and the answer to that is "yes, you can easily do that".

__________________________________________________

> I guess we're both right. I was talking about the "libraries" functionality

I'm confused, what functionality exactly are you saying is missing from the libraries functionality? That very example you link to shows one using the julia runtime via printing.

And yes, dynamic dispatch and dynamic code generation do work from libraries created by PackageCompiler.jl as well...

I would quibble with your use of the word "easily", but I consider it amazing that a language with such a complex runtime manages to expose an embeddable interface at all. Julia is an incredible language.

I guess the problem is that if you create a shared-library, yes, you can call it from Python, but you are now restricted to C datatypes, and don't get any of the type-conversion or type-proxying that the other bridges provide --- as far as I can see.

That said, I think the bridges are actually pretty developed compared to what exists between many pairs of high-level languages. They are clearly working for some people since you will find Python-wrapper packages for Julia code. I'm not so sure it's so important to have a single `.so` package. There are solutions which allow Python to be-in-charge and bundle Julia to various extents. There are at least:

1. https://github.com/jlapeyre/julia_project

2. Usage of Conda and https://github.com/JuliaPy/pyjuliapkg

So you can embed it in the same way you can embed e.g. python or octave. This is fine for some uses, but it is a bit invasive. How does it interact with multiple threads? That's a huge pain point in embedding python code...

edit: just found https://docs.julialang.org/en/v1/manual/embedding/#Thread-sa...

Ok, so, it's tricky, but at least you can start an interpreter per thread...

You can also run multi-threaded Julia from external threads. Julia unlike python doesn't have a GIL (or similar), and supports native multi-threading. As of 1.10 (I think, it might be 1.9), you can also add threads after startup (which is necessary for interfacing with spark or similar systems).

Yep that's right. Again though, PackageCompiler.jl does work quite well for creating .so / .dylib / etc shared libraries from julia code for more traditional library-style interaction, and is quite mature and widely used.

It produces quite big binaries, but the more involved and complicated your application is, the less that matters.

Reply on news.ycombinator.com