I completely agree -- it blows my mind that the BEAM/OTP is not more widely used. It is such a phenomenally well-designed system and it suits modern use cases perfectly.
It does have good FFI, like rustler for the elixir is great. It makes sure rust panics unroll into elixir exceptions and that takes some of the scariness of writing them.
That plus dirty schedulers means you don't have to be a good citizen and can write cpu intensive code in a faster language.
the main reference that shows up if you search for erlang + hpc is from 2008
https://www.researchgate.net/publication/221211398_High-perf...
I'm a Elixir and Erlang developer who is just starting to explore ML, but these libraries seem solid and well designed.
There is also the NX project (numerical elixir) which compiles tensors to GPU code making it much faster for those particular usecases. https://github.com/elixir-nx This has things like explorer which is elixir bindings to polars, livebook which is like jupyter notebooks (with enforced execution order thang god), aswell as some machine learning libraries in axon and scholar.
There has also been some work with libraries like flow and broadway which allow pipelines to be constructed with the usual syntaxt `step1 |> step2 |> step3` but to be executed across cores (or cluster).
I can imagine orgs using a combination of the above tools to do some interested distrubuted data pipelining and ML.
That said, it is a fantastic system for some problems and I really miss working with it.
We used it as the command and control language for these: https://www.icare-world.com/us/product/icare-eidon/
* Semi-embedded system that's used as an appliance, not a computer.
* Runs in an environment where people mostly don't want to mess with it. Might be days between reboots; might get turned off every night.
* Has to be pretty deterministic in terms of memory usage, response times, and so on.
* Recovers from something going wrong by going to a known-good state.
We didn't actually use any of the distributed aspects of it, but it worked out great for the other stuff.
Scripting, string munging, any sort of small task, anything which runs locally (unless it’s changed significantly erlang’s distribution story is the worst).