Zigler: Zig NIFs in Elixir
github.com
github.com
The BEAM VM (which is the thing that runs erlang / elixir / gleam / etc) has 3 flavors of functions.
- BIFs - Built-in functions, these are written in C and ship with the VM
- NIFs - Natively implemented functions, these are written in any language that can speak the NIF ABI that BEAM exposes and allows you to provide a function that looks like a built-in function but that you build yourself.
- User - User functions are written in the language that's running on BEAM, so if you write a function in erlang or elixir, that's a user function.
NIFs allow you to drop down into a lower level language and extend the VM. Originally most NIFs were written in C, but now a lot more languages have built out nice facilities for writing NIFs. Rust has Rustler and Zig now has Zigler, although people have been writing zig nifs for a while without zigler and I'm sure people wrote rust nifs without rustler.
The creator of Zigler has a talk from ElixirConf 2021 on how he made Zig NIFs behave nicely:
Discord is a big Erlang + Rustler user.
It's one less potential cause that might bring down the entire Erlang VM.
I wrote this recently about Go, but it equally applies to any Rust application that tries to recover from a panic.
This is based on the acknowledgment that if you have a large number of longer running processes at some point something will crash anyway, so you may quite as well be good at managing crashes ;-)
https://dev.to/adolfont/the-let-it-crash-error-handling-stra...
It's also not like there is much of a choice here. Unwinding across FFI boundaries (e.g. out of the NIF call) is undefined behaviour, so the only other option is aborting on panics.
I am still interested in the situation you observed.
This is kind of fascinating and seems worthy of more detailed study. I'm sure almost anything looks stable compared to javascript/python ecosystems, but would be interesting to see how other ecosystems with venerable old web-frameworks or solid old compression libraries compare. But on further reflection.. language metrics like "popularity" are also in danger of just quantifying the churn that it takes to keep working stuff working. You can't even measure strictly new projects and hope that helps, because new projects may be a reaction to perceived need to replace other stuff that's annoyingly unstable over periods of 5-10 years, etc.
Some churn is introduced by trying to keep up with a changing language, standard lib, or other dependencies, but some is just adding features forever or endlessly refactoring aesthetics under different management. Makes me wish for a project badge to indicate a commitment like finished-except-for-bugfixes.
Maybe it's the functionalness, maybe it's the problem domains, but a lot of the modules have clear boundaries and end up with pretty small modules where the libraries end up having a clear scope and a small code base that moves towards being obviously correct and good for most and then doesn't have much changes after that. It might not work for everyone, but most modules don't end up with lots of options to support all the possible use cases.
The underlying bits of OTP don't tend to churn too much either, so old code usually continues to work, unless you managed to have a dependency on something that had a big change. I recall dealing with some changes in timekeeping and random sources, but otherwise I don't remember having to change my Erlang code for OTP updates.
It helps that the OTP team is supporting several major versions (annual releases) simultaneously, so if there's a lot of unneccessary change, that makes their job harder as well as everyone else's.
https://elixir-lang.org/blog/2019/06/24/elixir-v1-9-0-releas...
It does get the occasional updates, but it's mainly related to developer tooling than language enhancements.
You can find libraries that haven't been updated in 10 years and yet are still the best solution.
If libraries get regular updates even if they are minor, it indicates they are in use. If they have inactive repositories and low hex.pm download numbers, they may have been abandoned which can mean you have to maintain it yourself in the future, or the people behind the library found it's not such a good idea after all. This doesn't have to be the case, which is why I asked.
I do think more libraries should give that little "We're still maintained" notice as people not totally ingrained in this might not realize. To some, the fact that there have been no issues reported now that we're on OPT 27 and Elixir 17 would be an indicator that all is well.
Something about immutability and the structure of Elixir leads to surprisingly few bugs.
With data structures that have some definite behavior unless someone finds a defect there isn’t going to be much activity.
Yes, it's Apache 2.0
There's another option and that's setting up an Erlang node in the other language. The Erlang term format is relatively straightforward. But I'm honestly not sure of the benefit of a node versus just using a port.
- can "easily" send beam terms back and forth
- if you want it to be os-supervised separately (systemd, kubernetes, e.g.)
- pain in the ass
Port:
- easy
- usually the only choice if you're not the software author
- really only communicates via stdio bytestreams
- risk of zombies if... Iirc the stdout is not closed properly?
- kind of crazy how it works, Erlang VM spawns a separate process as a middleman
It's not impossibly large but it's not something one does on a lark either; if there isn't support in your language already it's hard to justify this over any of the many, many message busses supported by both Erlang and other languages that don't have so many requirements.
If you've got some mathematical/crypto function, chances are you don't want that to go through a command queue to an external port, because that's too much overhead. If it's a many round crypto function like bcrypt or something, you do need to be a bit careful doing it as a NIF because of runtime. But you wouldn't want to put a sha256 through an external program and have to pass all that data to it, etc.
Something that you might actually want queueing for and is likely to have potential for memory unsafety like say transcoding with ffmpeg, would be a good fit as an external Port rather than a NIF or a linked in Port driver.
We do have IPC handles that could enable this over, say, ports, but then there's a whole other discussion on pointers vs ipc handles
Forgive me if I'm mixing up my terminology it's been a bit since I have poked at Elixir.
https://www.erlang.org/doc/apps/erts/erl_nif#enif_schedule_n...
After all, many of the BIFs have been replaced internally by NIFs
And there's this, which would scare me:
https://erlang.org/documentation/doc-15.0-rc3/erts-15.0/doc/...
Here’s a link I found talking about using the dirty scheduler with Rust(ler): https://bgmarx.com/2018/08/15/using-dirty-schedulers-with-ru...
I always find actually doing that — and then maintaining the results over time — to be quite painful: you don't get syntax highlighting inside the string; you can no longer search your worktree reliably using extension-based filtering; etc.
I personally find the workflow much more sane if/when you just have a separate file (e.g. `foo.zig`) for the guest-language code, and then your host-language code references it.
It's certainly possible to get syntax highlighting on the embedded code, but you'll need to work with your syntax highlighter; it certainly helps if you're not the only person using it.
But then again, I worked without syntax highlighting for years, so I'm happy when it works, but when it doesn't, I'm ok with that too.
That being said, you can get IDE language support for embedded code if you use eMacs or vim (and probably other editors as well). As I mentioned I still vastly prefer separating it personally, especially if you don’t necessarily expect your Python or Typescript programmers to be knowledgeable about Zig (or C).
Also, I'm not sure why it's not better documented in Zigler, but you can also write the code in a separate file just fine.
> Syntax highlighting here can work correctly, actually.
Highlighting shown here in the 2021 ElixirConf talk posted elsewhere in the comments:
https://youtu.be/lDfjdGva3NE?t=2064
> I'm not sure why it's not better documented in Zigler
Here's the docs for it (though buried in the 'advanced' section)
https://hexdocs.pm/zigler/Zig.html#module-importing-external...
But, if all you do is write elixir wrappers around the zig function, to completely hide the foreign language functions, keeping both the wrapper and implementation in the same file, even if two different languages doesn't seem horrible, but again, keeping them in two file doesn't seem like a huge difference too
I think its really a matter of taste, both options viable
Isn’t that essentially any web application?
I wish zig got more use and attention in the Erlang ecosystem, but rustler seems more popular.
I can appreciate Zig for entire projects that would otherwise be written in C, but for the lengths of code that make sense for a NIF as opposed to a port, Zig seems like a strange point of failure to add to my system. If it's simple enough that I can be confident in my flawless manual memory management, I'd just use C, and for anything else, Rust is the far safer choice.
Sounds interesting, is it open source? I am interested in seeing how the code layout looks like when mixing Zig and Elixir
// the_nif.zig
fn init_imp(
env: ?*erl.ErlNifEnv,
argc: c_int,
argv: [*c]const erl.ERL_NIF_TERM,
) !erl.ERL_NIF_TERM {
if (argc != 0) {
return error.BadArg;
}
return try helpers.make("Hello world");
}
export fn media_tools_init(
env: ?*erl.ErlNifEnv,
argc: c_int,
argv: [*c]const erl.ERL_NIF_TERM,
) erl.ERL_NIF_TERM {
return init_imp(env, argc, argv) catch |err|
return helpers.make_error(env, err);
}
var funcs = [_]erl.ErlNifFunc{ erl.ErlNifFunc{
.name = "init",
.arity = 1,
.fptr = media_tools_init,
.flags = erl.ERL_NIF_DIRTY_JOB_CPU_BOUND,
} };
var entry = erl.ErlNifEntry{
.major = erl.ERL_NIF_MAJOR_VERSION,
.minor = erl.ERL_NIF_MINOR_VERSION,
.name = "Elixir.MediaTools.Stream",
.num_of_funcs = funcs.len,
.funcs = &funcs,
.load = load,
.reload = null,
.upgrade = null,
.unload = null,
.vm_variant = "beam.vanilla",
.options = 0,
.sizeof_ErlNifResourceTypeInit = @sizeOf(erl.ErlNifResourceTypeInit),
.min_erts = "erts-10.4",
};
export fn nif_init() *erl.ErlNifEntry {
return &entry;
}
# the_exlixir_file.ex
assert "Hello world" == MediaTools.Stream.init()
I wrote a piece back in 2022, but things evolved a lot since then: https://www.kuon.ch/post/2022-11-26-zig-nif/
NIF responsibly. :)
There was a popular motivational speaker in the 80's and 90's named Zig Ziglar[0]. He was influential on Tony Robbins and his career.
Just shows even how a randomly generated name [1] may not be so unique!
[0] https://en.wikipedia.org/wiki/Zig_Ziglar [1] https://en.wikipedia.org/wiki/Zig_(programming_language)#Ori...
1. More accurately, NIFs sre BEAM's take on FFI functions, and Elixir is a BEAM language.
Elixir sigils also allow multiple characters in the name, but chars after the first must be upper case, according to the docs.
So for Elixir, it would have to be something like ~zIG
> Custom sigils may be either a single lowercase character, or an uppercase character followed by more uppercase characters and digits.
I wonder if the Zig code can be not written inline, as an option. With anything larger than a few lines, I'd want syntax highlighting, LSP support, navigation, etc. It's easier to achieve with one language per file.