Embedding Python in Elixir, it's fine | Hacker News Reader

Embedding Python in Elixir, it's fine

dashbit.co

308 pointsarathunku64 comments

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood.

For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's running on the same OS process as your Elixir app, you bypass the failure recovery that makes an Elixir/BEAM application so powerful.

Normally, an Elixir app has a supervision tree that can gracefully handle failures of its own BEAM processes (an internal concurrency unit -- kind like a synthetic OS process) and keep the rest of the app's processes running. That's one of the big selling points of languages like Elixir, Erlang, and Gleam that build upon the BEAM architecture.

Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

There are cases when NIFs are super helpful (for instance, Rustler is a popular NIF wrapper for Rust in Elixir), but you have to architect around the fact that it could take down the whole app. Using Ports (Erlang and Elixir's long-standing external execution handler) to run other native code like Python or Rust is less risky in this respect because the non-Elixir code it's still running in a separate OS process.

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

I'm more of a Python and C# kind of guy, so Elixir never really hit the itch for me, but Gleam definitely does. One of these days I'll take a crack to see how I can use Gleam with Phoenix.

I’ve been mostly in Python, C# and C++ for the past decade or so but got into Elixir as my first functional language. Never got comfy with the syntax but dig how everything flows. Looking forward to digging into Gleam.

If you liked Elixir but found it too "exotic" you may find F# enjoyable instead - it's a bit like Elixir but with a very powerful, gradually typed and fully inferred type system and has access to the rest of .NET. Excellent for scripting, data analysis and modeling of complex business domains. It's also very easy to integrate a new F# project into existing C# solution, and it ships with the SDK and is likely supported by all the tools you're already using. F# is also 5 to 10 times more CPU and memory-efficient.

(F# is one of the languages Elixir was influenced by and it is where Elixir got the pipe operator from)

Been on the list for years — I’ll check it out.

I’d recommend to first see if you can use a full-Gleam solution (like wisp/lustre) if it’s a greenfield project – interop is of course possible but can sometimes be a bit unpleasant due to the difference in data structures (Elixir structs va Gleam records) and inability to use Elixir macros directly from Gleam, which are heavily used by projects like Phoenix and Ecto.

My current favorite language, just no time to finish my gleam projects.

Ah, projects. Ain’t it always the way.

I’ve been eyeing gleam as my next language to learn. Lots to like about it for sure, and I have always like the idea of OTO but never had an opportunity to tinker with it.

One possibility for production use (in case there is a big value) is to split the nodes into one "front" node which requires strong uptime, and a "worker" node which is designed to support rare crashes gracefully, in a way that does not impact the front.

This is what we use at https://transport.data.gouv.fr/ (the French National Access Point for transportation data - more background at https://elixir-lang.org/blog/2021/11/10/embracing-open-data-...).

Note that we're not using Pythonx, but running some memory hungry processes which can sometime take the worker node down.

Do any of them communicate with the BEAM? There used to be a Go based implementation of the BEAM that allowed you to drop-in with Go, I have to wonder if this could be done with Python so it doesn't interfere with what the BEAM is good that and lets Python code remain as-is.

There are several libraries that allow a Python program to communicate with an Erlang program using Erlang Term Format and such.

This approach targets more performance-sensitive cases with stuff like passing data frames around and vectors/matrices that are costly to serialize/deserialize a lot of the time.

And it seems to make for a tighter integration.

> Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

What's the Elixir equivalent if "Pythonic"? An architecture that allows a NIF to take down your entire supervision tree is the opposite of that, as it defeats a the stacks' philosophy.

The best practice for integrating Python into Elixir or Erlang would be to have an assigned genserver, or other supervision-tree element - responsible for hosting the Python NIF(s), and the design should allow for each branch or leaf of that tree to be killed/restarted safely, with no loss of state. BEAM message passing is cheap

That's the thing though: a NIF execution isn't confined to the the BEAM process by its nature. From the Erlang docs:

> As a NIF library is dynamically linked into the emulator process, this is the fastest way of calling C-code from Erlang (alongside port drivers). Calling NIFs requires no context switches. But it is also the least safe, because a crash in a NIF brings the emulator down too. (https://www.erlang.org/doc/system/nif.html)

The emulator in this context is the BEAM VM that is running the whole application (including the supervisors).

Apparently Rustler has a way of running Rust NIFs but capturing Rust panics before they trickle down and crash the whole BEAM VM, but that seems like more of a Rust trick that Pythonx likely doesn't have.

The tl;dr is that NIFs are risky by default, and not really... Elixironic?

> The emulator in this context is the BEAM VM that is running the whole application (including the supervisors)

You are correct - one could still architect is such that the genserver hosting the NIF(s) run in a separate process/VM/computer in the same cluster since message passing is network-transparent, though inter-host messages have higher latencies.

> an unhandled exception in Pythonx would take down your whole OS

Is there a class of exceptions that wouldn't be caught by PythonX's wrapper? FTA (with emphasis added):

> Pythonx ties Python and Erlang garbage collection, so that the objects can be safely kept between evaluations. Also, it conveniently handles conversion between Elixir and Python data structures, bubbles Python exceptions and captures standard output.

And...

> Rustler is a popular NIF wrapper for Rust in Elixir

From Rustler's Git README:

> The code you write in a Rust NIF should never be able to crash the BEAM.

I haven't used Rustler, Zigler or PythonX (yet), so I'm genuinely asking if I'm mistaken in my understanding of their safety.

I love to see "well-known" people in the Elixir community endorsing and actively developing that kind of approach. Our VM and runtime does so much and is so well suited to orchestrating other languages and tech that it sometimes feels there's a standard track and an off-road track.

The difference between an off-road "sounds dangerous" idea and its safe execution is often only the quantity of work but our runtime encourages that. Here, it's a NIF so there's still a bit of risk, but it's always possible to spawn a separate BEAM instance and distribute yourself with it.

Toy example that illustrates it, first crashing with a NIF that is made to segfault :

  my_nif_app iex --name my_app@127.0.0.1 --cookie cookie -S mix
  iex(my_app@127.0.0.1)1> MyNifApp.crash
  [1]    97437 segmentation fault

In the second example, we have a "SafeNif" module that spawns another elixir node, connects to it, and runs the unsafe operation on it.

  my_nif_app iex --name my_app@127.0.0.1 --cookie cookie -S mix
  iex(my_app@127.0.0.1)1> MyNifApp.SafeNif.call(MyNifApp, :crash, [])
  Starting temporary node: safe_nif_4998973
  Starting node with: elixir --name safe_nif_4998973@127.0.0.1 --cookie :cookie --no-halt /tmp/safe_nif_4998973_init.exs
  Successfully connected to temporary node
  Calling MyNifApp.crash() on temporary node
  :error
  iex(my_app@127.0.0.1)2>

Thankfully Python, Zig and Rust should be good to go without that kind of dance :) .

Its a neat way to do it - spin a temporary one, which can crash all it wants without affecting the other nodes. Fits like a glove to BEAM.

Great and informative article. Also nice to get an explicit mention that this isn't just a subprocess call, but running in the same process.

The only thing I'd would have like to see in added would be calling a function defined in Python from Elixir, instead of only the `Pythonx.eval` example.

The `%{"binary" => binary}` is very telling, but a couple of more and different examples would have been nice.

Other commenters have already pointed out the safety implications of using NIFs for this. There are, however, other downsides worth considering:

- The Erlang VM scheduler can't preempt a NIF, so a long-running Python call risks hanging the VM. This is a non-issue for ports, since Python's running in a separate OS process. A NIF can mitigate this by spawning an OS thread and yielding until it finishes; ain't clear if that's what this library is doing.

- The article already mentions that the GIL prevents concurrent Python execution, but this would also be a non-issue for ports, since the Erlang caller would just spin up multiple Python interpreters. Does Python allow multiple interpreters per (OS) process, like e.g. Tcl does? If so, then that'd be a possible avenue for mitigating this issue while sticking with NIFs.

I would have guess the builders of this would have mitigated this problem by running the python in a thread? That won't hang the VM (or cause a segfault at the 1ms boundary). It might cause OS starvation in extreme cases, but you'd have to be really extreme.

There are dedicated "dirty" schedulers for long-running NIFs that by nature cannot be preempted, to avoid interfering with BEAM process scheduling : https://www.erlang.org/doc/apps/runtime_tools/scheduler.html

These are excellent points.

Really glad to see this, Elixir has languished in the AI wars despite being a better fit than JavaScript and Python.

Forgive some ignorance on this; why is Elixir a better fit for AI than Python or JavaScript? I'm not disagreeing, I've just never heard that, I didn't think that Elixir had good linear algebra libraries like NumPy.

It does now with Nx

Sorry, I should have been more explicit: better for on the user facing implementation side (concurrency, streaming data, molding agent state, etc) vs the training side of things. If that makes sense.

Ah, fair enough. I've not done much with Elixir but I have done a fair amount with Erlang and you certainly don't need to sell me on how great it is for concurrency and distributed stuff.

I’ve been actively using elixir for ML at work, and I would say it’s a solid choice.

The downside - unfortunately while bumblebee, Axon, and Nx are libraries that seem to have a fantastically engineered base most of the latest models don’t have native elixir implementations yet and making my own is a little beyond my skill still. So a lot of the models you can easily run are older.

But the advantages - easy long running processes, great multiprocessing support, solid error handling and recovery - all pair very well with AI systems.

For example, it’s very easy to make an application that grabs files, caches them locally, and runs ML tasks against them. You can use process monitoring and linking to manage the locally cached files, and there’s no runtime duration limit like you might hit in a serverless system like lambda. Interprocess messaging means you can easily run ML in a background task and stream results asynchronously to a user. Additionally, logs are automatically streamed to the parent process and it’s easy to tag logs with process metadata, so tracking what is going on in your application is dead simple.

That’s basically a whole stack for a live ML service with all the difficult infrastructure bits already taken care of.

I love the initial decision to grow Elixir's ML foundations from scratch, but I also love that we now have a really ergonomic way to farm out to the fast-moving python libraries

> Also, it conveniently handles conversion between Elixir and Python data structures, bubbles Python exceptions and captures standard output

Sooo nice

At first read this seems really promising. Getting into Elixir/Erlang ecosystem from Python has seemed too hard to take the time. And when there I wouldn't be able to leverage all the Python stuff I've learned. With Pythonx gradual learning seems now much more achievable.

It wasn't mentioned in the article, but there's older blog post on fly.io [1] about live book, GPUs, and their FLAME serverless pattern [2]. Since there seems to be some common ground between these companies I'm now hoping Pythonx support is coming to FLAME enabled Erlang VM. I'm just going off from the blog posts, and am probably using wrong terminology here.

For Python's GIL problem mentioned in the article I wonder if they have experimented with free threading [3].

[1] https://fly.io/blog/ai-gpu-clusters-from-your-laptop-liveboo...

[2] https://fly.io/blog/rethinking-serverless-with-flame/

[3] https://docs.python.org/3/howto/free-threading-python.html

FLAME runs the same code base on another machine. FLAME with Pythonx should just work. FLAME is a set of nice abstractions on top of a completely regular Erlang VM.

Chris Grainger who pushed for the value of Python in Livebook has given at least two talks about the power and value of FLAME.

And of course Chris McCord (creator of Phoenix and FLAME) works at Fly and collaborates closely with Dashbit who do Livebook and all that.

These are some of the benefits of a cohesive ecosystem. Something I enjoy a lot in Elixir. All these efforts are aligned. There is nothing weird going on, no special work you need to do.

Yeah, looks like it works fine, here's an example: https://pastebin.pl/view/a10aea3d

I'll add: FLAME is probably a great addition to pythonx. While a NIF can crash the node it is executed on, FLAME calls are executed on other nodes by default. So a crash here would only hard-crash processes on the same node (FLAME lets you group calls so that a flame node can have many being executed on it at any time).

Errors bubble back up to the calling process (and crash it by default but can be handled explicitly), so managing and retrying failed calls is easy.

That's a really good option - neatly sidesteps the risk of a NIF crash with practically no extra code.

You do still need some infrastructure - the Flame LocalBackend is for dev mostly and I'm pretty sure it just runs in the same vm as the parent.

But yeah if you're doing ML tasks it makes a lot of sense to farm those out to beefier or GPU-equipped nodes anyway so at that point it's just a natural synergy, AND you get the crash isolation.

Well this seems nice and easy. Thank you for the example. There's local, Kubernetes and fly.io support for FLAME (that I found after short search). I envision running main Erlang VM on a light weight server continuously, and starting beefier machines for Python tasks as needed.

Elixir has some features I wish Python had:

- atoms

- everything (or most things) is a macro, even def, etc.

- pipes |>, and no, I don't want to write a "pipe" class in Python to use it like pipe(foo, bar, ...). 90% of the |> power comes from its 'flow' programming style.

- true immutability

- true parallelism and concurrency thanks to the supervision trees

- hot code reloading (you recompile the app WHILE it's running)

- fault tolerance (again, thanks for supervision trees)

Mix is also so much better than anything python has to offer in terms of build/dependency tooling.

Jupyter might have fixed this now because it’s been a while since I used it, but Mix.install inline in Livebook (or any CLI script) is so much nicer than how installing Python dependencies in notebooks was last time I did that too.

uv for Python is a game changer, better than anything else out there and solves a lot of the core problems with pip/venv/poetry/pyenv (the list goes on).

I feel like you can write some variant of this comment every few years and just add the previous "best" to the front of the stack of things it's better than.

It’s true - people were saying that Poetry solves these problems for ages. Maybe uv does? I’ll wait and see.

This time they've got it for sure!

You're not wrong, this is the nth iteration of python tools that try to solve all the problems of what came before, including whatever the n-1th iteration introduced.

That said, in my personal experience with uv, it solves nearly all of the problems I've come across that were created by other python package management tools. It seems to have been very thoughtfully designed and I think there's a strong chance it'll become the standard, and that there won't need to be more standards after this. We'll see!

Coming from Erlang, I think macros are one of the things I'm ambivalent about in Elixir. There are a bunch of actual improvements besides just the syntax itself in Elixir, like string handling, but things like macros in Ecto ... not yet a fan of that.

> ..but things like macros in Ecto....

Well, The whole language itself is built on macros. The following series of articles certainly helped me stop worrying and love the macros..

https://www.theerlangelist.com/article/macros_1

Some interesting insights from the article: "Elixir itself is heavily powered by macros. Many constructs, such as defmodule, def, if, unless, and even defmacro[1] are actually macros...."

[1] https://github.com/elixir-lang/elixir/blob/v1.18.2/lib/elixi...

In practice you don't really need to write macros if you don't make libraries.

You can abuse the '>>' notation in python for pipes (or you could use |, I suppose), but you'll have to deal with whitespace shenanigans. I'm also not entirely sure about the order of evaluation. And you'll need to do partial function application by hand if you want that (though it is possible to write a meta function for that).

So one could write

    class Piped:
        def __init__(self, value):
            self.value = value
    
        def __or__(self, func):
            return Piped(func(self.value))
    
        def __repr__(self):
            return f"Piped({self.value!r})"
    
    Piped('test') | str.upper | (lambda x: x.replace('T', 't')) | "prefix_".__add__ # => prefix_tESt

but whether that is a good idea is a whole different matter.

Apache Beam in Python does this, with code like

    counts = (
        lines
        | 'Split' >> (
            beam.FlatMap(
                lambda x: re.findall(r'[A-Za-z\']+', x)).with_output_types(str))
        | 'PairWithOne' >> beam.Map(lambda x: (x, 1))
        | 'GroupAndSum' >> beam.CombinePerKey(sum))

I'm not sure how I feel about it, other than the fact that I'd 100x rather write Beam pipelines in basically any other language. But that's about more than syntax.

The part of Elixir pipes that people overlook is the fact that, as a native, heavily used part of the language, the entirety of the standard library and most third party libraries are written with their behavior in mind. The first argument (the pipe target) will be the argument you want piped in faaaar more often than in languages where pipes are tacked on later.

As someone very involved in Elixir and who used to do a lot of Python this seems very practical for me. I'm actually even more interested in that Fine library for making C++ NIFs easy. That seems ridiculously valuable for removing hurdles to building library bindings.

I feel like this project and blog post was made specifically for me. Can't wait to use this, thanks!

I love this, I've primarily been working in Elixir for a few years now and this is neat to see!

Looks like a very cool way to interop with Python from Elixir without maintaining a separate Python stack (which is a PITA)!

I was super excited until I read:

"...if you are using this library to integrate with Python, make sure it happens in a single Elixir process..."

What’s the problem? You plan on running multiple instances of you Elixir program?

I'm using Python script in my Phoenix app (not Livebook). I hoped Pythonx would solve all the issues associated with System.cmd, but as you can imagine, I have more than one user.

“Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should”

just kidding, this is pretty cool.

yes

Elixir is just Lisp with a facelift[1], and lisps can be built on Python[2]. It stands to reason that an elixir-like can be built on Python too, so you could embed the Python runtime in Elixir but Elixir-likes are used to code for both.

1: https://wiki.alopex.li/ElixirForCynicalCurmudgeons

2: https://hylang.org/

In a way Python is a bad Lisp, still looking forward that catches up in native code compilation and multiline lambdas.

Could be better, but that is what mainstream gets.

We can sort of make a shitty multi-line lambda in Python by making a dummy function called progn, and using it, like this:

  >>> def progn(*args):
  ...   if args:
  ...     return args[-1]
  ...   else:
  ...     return None
  ...
  >>> fun = lambda x : progn(print('abc'),
  ...                        print(x),
  ...                        print('def'))
  >>>
  >>> fun(42)
  abc
  42
  def

The operating environment of the BEAM is what's great about elixir. Hy still has the GIL.

Reply on news.ycombinator.com