Elixir and Machine Learning: Nx v0.1 released

Adding to this, Jose was also streaming his solutions on Twitch https://www.twitch.tv/josevalim/videos

I was learning Elixir through advent of code, and this was an invaluable resource for getting my head around an more functional (or Elixir-y?) style of problem-solving.

I just want to thank you for your incredible work. I'm a fellow countryman migrating a Python program that fetches financial data from a large amount of sources, that's become very hard to maintain, to Elixir. The BEAM fits my use case perfectly, and Elixir has been an absolute joy to work with and debug.

Off-topic: hi Jose, what's your expectation on how much more raw performant Erlang can become in the next 1-2 years?

PHP ~3x in performance from 5.x to 7.x. LuaJIT is even faster over vanilla Lua.

There were multiple attempts at creating an Erlang JIT for well over a decade now and last year BeamAsm was released but in everyday use it seems to net about a 10-15% boast.

I ask because it'd be great if we didn't have to create so many Rust NIFs in order to address Erlang raw slowness.

Curious what your take is on speeding up raw Erlang (please note, I'm not talking about concurrency & I do truly love Exilir/Erlang/OTP)

Very impressive library! Is it still in the developmental stage, or do you know of anyone using it for real yet?

Also, I love livebook! I keep a tab with it open and it has replaced my use of iex. Can you talk a little bit about how you decided to embark on building that, too? An ML library and framework is a big enough job already, how did you decide to add Livebook to the list?

Are there any specific advantage that Elixir being on BEAM VM brings to Machine Learning and Data Science? I am assuming BEAM VM's actor model fits well with the GPU / Tensor usage ? ( I have zero knowledge on ML and Data Science )

Personally I think this is a great move from Elixir community and especially coming directly from the creator. Some people may be dismissive ( as with at least half of HN ) But I see this with Ruby where there is a chicken and egg problem. The community stuck with Web or Rails Development only. And no one has incentive to kickstart the languages ecosystem in different directions.

So instead of Elixir being another language for Web / Phoenix, it could now ( or in the future ) be used as ML with some level of official support.

Hi José, not Nx related but would you consider adding a RSS feed to the Dashbit blog?

You sir, are a legend.

I was particularly delighted to see how you built `defn` on top of existing metaprogramming functionality. This kind of thing has lead me to think of Elixir as a Lisp in Ruby's clothing running on the BEAM. :) I hope that's not too inaccurate. Were there any delightful "aha!" moments you got while building this?

Thank you so much for all of this great work. What was the inspiration for this direction? Was it a personal itch? Or something else?

I think right now, this is for people who already know and need the advantages of Erlang/Elixir and want to be able to do numerical computing without having to resort to a NIF or ports. Numerical computing has been one of the big weaknesses with the BEAM platform, even though it is among the first things people think about when they find out about how BEAM does concurrency. I can tell you that many Elixir developers are quite excited about this. It is definitely scratching an itch.

I don't know how this is emerging. If I take a stab at it, BEAM/OTP/Erlang/Elixir is very good at coordinating among many concurrent processes as well as handling failures that come with it. There really isn't any other language platform that does that as well, including Python. (Although there is interest in recreating those advantages in Rust). The distributed computing is built on top of those concurrency primitives. So when one thinks about adding numerical computing into the mix, I think of:

- Broadway (an Elixir library) that handles workflows from unreliable data sources. It is already in production use to handle massive, distributed web scraping operations, so embedding a data transform in there with numeric computing will have good synergies

- IoT applications, both at the edge with Nerves, and with the command/control for IoT. Each of the IoT devices are unreliable (power and network), and OTP already has great facility for that. Being able to do numeric computing at the edge is a thing.

That's an excellent question!

We are not trying to make Elixir the new AI language, but one possible language for AI. @hosh covered some of the use cases in his reply and I can add two more insights:

* Lately we have seen functional ideas brought into Python through projects like JAX and Thinc.ai and I think exploring those concepts within a functional language is both interesting and exciting (and we are not the only ones doing so!)

* Besides IoT and data pipelines, there is potential in mixing some of the current ML trends, such as Distributed and Federated Learning, with the capabilities of the platform

We likely aren't much attractive to Python developers right now, unless you are interested in contributing/designing ML tooling (for learning, hobby, or professional reasons) or you have a use case that suits one of the mentioned strengths of the platform.

This a parallel and independent ecosystem for numerical computing with equivalent libraries:

* pandas - explorer

* numpy - nx

* statsmodel - Not sure this exists yet

The article answers this pretty well.

However, our goals are also to:

* make Elixir a suitable platform for new Machine Learning developments

* fully leverage the power provided by the platform Elixir runs on, the Erlang VM

* provide consistency and stability, especially when working on a domain that is still actively evolving

For those reasons, we chose to invest on Nx as its own foundation, agnostic to any particular framework. The road is definitely longer but we also believe the pay-off will be higher too!

> I really like python

Give elixir a shot. I used to use python (though not as much as I used ruby) and I will never go back to python.

You are correct. Currently we are waiting for the issue to be tackled upstream (both on LibTorch and Google's XLA). Someone could also implement a specific Nx-backend for Apple. It should actually be fun but not high priority at the moment.

Well there isn't an Nx.dev plugin for Elixir, is there? Nx for Elixir is a nice shorthand for Numerical Elixir. Nx.dev just looks like a consulting company jumping on a short domain when the .dev TLD became available and then using it for a project later. The nx.dev site was only launched in 2019, and then as an Angular dev tool, and as an Elixir dev I hadn't heard of it until after Nx elixir existed, which seems to have started some time in mid to late 2020.

[1]https://web.archive.org/web/20190501153325/https://nx.dev/