Bumblebee: GPT2, Stable Diffusion, and More in Elixir
news.livebook.dev
news.livebook.dev
The ergonomics of Bumblebee are so perfect.
I've also been doing the fastai course, where you learn Gradio and pytorch.
Python has such a messy library story. I'm not a python developer, and coming into this ecosystem and trying to make things work with pip, conda, docker, etc. It's a mess.
I like Gradio, and built a few small apps, but it is still messy compared to Bumblebee.
Livebook + Bumblebee is magical. I'm productive in an instant, and the opportunity to build with Elixir and Phoenix makes this so exciting.
I'm blown away.
I don't love the node/npm library ecosystem entirely, but it seems like at least the tools are consistent: I can generally run npm or yarn or pnpm and it will work with the same package.json file. There are often weird issues with ES6 or common JS, but generally the libraries have a fix when that is an issue.
With python, when I look at a new repository, I'm always trying to figure out, ok, does it have a requirements.txt (which means pip, right?). Or, do I need to use conda and activate it? Does anyone still use virtualenv (and why not)? And, often the requirements listed in the repository seem to only work in certain contexts. I've tried normalizing it all using a docker wrapper, but, especially for ML work, I'm then fighting with the CUDA libraries, and running against the correct GPU host libraries, etc. It feels so complicated and requires you to have been in the ecosystem for years to have the intuition on where to go next.
And, it probably makes things even more complicated that my host system is nixos: I don't think python ever considered immutable filesystems in the worldview.
I was hoping that the fastai course would smooth out some of this for me, but when I started trying to run on my own hardware, it quickly fell apart. This has always been the case for me with ML: trying to just get something running so I can experiment is such a battle that I give up.
Elixir just seems like it has a more sane approach for libraries and generally seems to work the right way. That's a good foundation for progress for me.
You may checkout https://news.ycombinator.com/item?id=34008201
It is late here but feel free to drop questions and I will answer them when I am up. Meanwhile I hope you will enjoy the content we put out: the announcements, example apps, and sample notebooks!
Maybe a more serious question: is anyone using Elixir ML in production? I'm absolutely gobsmacked at the quantity and quality of development effort that's gone into it (and use Livebook daily, though not for ML stuff). It's clearly a major focus for the team. I'm wondering if it's ready for production adoption, and if so, if anyone has used it "in anger" yet.
Checkout this recent talk about how https://www.amplified.ai/ moved from Python to an Elixir ML stack: https://www.youtube.com/watch?v=Y2Nr4dNu6hI
We were able to completely eliminate a few python services and consolidate to all Elixir for ETL and ML.
If I have a Huggingface model that I've finetuned, can I load it using Bumblebee? I fine-tuned ConvNext and changed it into a multi-label classifier and saved it as a PyTorch model. It works great but being able to use it in LiveBook instead of Jupyter Notebook would be fantastic.
I think I'd have to convert the format, but what then?
Alternatively to EXLA/Torchx, any thoughts on supporting an ML compiler frontend like Google IREE/MLIR by generating StableHLO or LinAlg? This could pave the way towards supporting multiple hardware targets (Vulkan-based, RVV-based, SME-based etc.) with minimal effort from the framework.
I think it would be great to see!
How easy would it be to support OpenAI's new Whisper transcription model in Bumblebee?
Can you provide some thoughts on the benefits of doing ML on Elixir vs. Python? Is the benefit the language semantics? Is it much easier to get distributed work in Elixir ML vs Python ML? Are the tools better/smoother? Are there ML ops improvements? Perhaps there’s a blog post I missed :)
In a nutshell, there has been trends in Python (such as JAX and Thinc.ai) that argue functional programming can provide better abstractions and more composable building blocks for deep learning libraries. And I believe Elixir, as a functional language with Lisp-style macros, is in an excellent position to exploit that - as seen in "numerical definitions" which compile a subset of Elixir to the CPU/GPU.
I also think the Erlang VM, with its distribution and network capabilities, can provide exciting developments in the realm of federated and distributed learning. We aren't exploring those aspects yet but we are getting closer to having the foundation to do so.
Regarding ML ops, I believe one main advantage is explained in this video announcement. When deploying a ML model with Nx, you can embed the model within your applications: you don't need a 3rd-party service because we batch and route requests from multiple cores and multiple nodes withing Erlang/Elixir. This can be specially beneficial for projects like Nerves [5] and we will see how it evolves in the long term (as we _just_ announced it).
Finally, one of the benefits on starting from scratch after Python has paved the way is that we can learn from its ecosystem and provide a unified experience. You can think of Nx as Numpy+JAX+TFServing all in one place and we hope that doing so streamlines the developer experience. This also means libraries like Scholar [2] (which aims to serve a similar role as SciPy) and Meow [3] (for Genetic Algorithms) get to use the same abstractions and compile to the CPU/GPU. The latter can show an order of magnitude improvement over other currently used frameworks [4].
[0]: https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... [1]: https://dashbit.co/blog/elixir-and-machine-learning-nx-v0.1 [2]: https://github.com/elixir-nx/scholar/ [3]: https://github.com/jonatanklosko/meow [4]: https://dl.acm.org/doi/10.1145/3512290.3528753 [5]: https://www.nerves-project.org/
Has anyone switched to Elixir from Go for writing web apps? What has been your experience like?
Elixir is quirky and the functional aspects especially take getting used to (eg the learning guides say you should rarely write for loops), but the runtime and some of these amazing tools really show the potential. I don’t know if I’ll ever use it professionally (I hope so, honestly) but I think I’ll learn something that’ll make me better regardless.
LiveView has such a unique and refreshing take on web-dev (I say having never written any significant amount of HTML), that seems to make it behave much more like a local application. And LiveBook builds on that to be a next-gen alternative to Jupiter notebooks. I can totally imagine live book expanding into a more general purpose interactive playground type tool.. good for internal tools and dashboard and development experiments… almost a WebUI alternative to a REPL. I feel like all that’s missing is being able to inject/attach to an existing VM.
That's basically how my team have been increasingly using it. Simply connect Livebook to a locally running Phoenix project and you have a Livebook REPL into your server. When you're dealing with complex data, pulling from different sources and have to build up a bunch of context before you iterate on a function it's super useful to be able to break up that code into chunks, take form inputs[0] along the way and document any quirks. We keep a bunch of livebooks committed in the repo to help debug and iterate on the more complex parts of our codebase.
It was a new client and a short project but since then they closed a couple years contract to a new project, Livebook is indeed magic.
If you mean connecting to a running node, that’s already possible! Here’s an example on fly.io’s infra: https://fly.io/docs/elixir/advanced-guides/connect-livebook-...
0: https://drive.google.com/file/d/1Mw_NEER4VzA1qhFIq6WH9PYLLbN...
As a Jupyter notebook user who has not tried Livebook (yet) i can summarize after watching:
- in Jupyter notebook state is hard to reproduce because execution can go back and forth between cells and that is not tracked. in Livebook execution is always top to bottom (still interactive and you can have branches)
- built-in package management
- multiplayer (the same notebook can be accessed and manipulated by multiple users)
- easier version control (a subset of markdown)
- ...
I think the biggest difference that'll trip you up coming from Jupyter is that Livebook enforces linear execution. You can't arbitrarily run cells in any order like you can in Jupyter - if you change an earlier cell all the subsequent cells have to be run in order. The only deviation from this is branches which allow you to capture the state at a certain point and create a new flow from there on. There's a section in [1] that explains how branching works and how you can use it when training models.
The other difference is that if you do something that crashes in a cell, you'll lose the state of the entire branch and have to rerun from the beginning of the branch. Iirc if you stop a long running cell, that forces a rerun as well. That can also be painful when running training loops that run for a while, but there are some pretty neat workarounds you can do using Kino. Using those workarounds does break the reproducibility guarantees though.
Personally while building NN models I find that I prefer the Jupyter execution model because for NNs, rerunning cells can be really time-consuming. Being able to quickly change some variables and run a cell out of order helps while I'm exploring/experimenting.
Two things I love about Livebook though are 1) the file format makes version control super easy and 2) Kino allows for real interactivity in the notebook in a way that's much harder to do in Jupyter. So in Livebook you can easily create live updating charts, images etc that show training progress or have other kinds of interactivity.
If you're interested to see what my model training workflow looks like with Livebook (and I have no idea if it's the best workflow!), check out the examples below [1][2]. Overall I'd say it definitely works well, you just have to shift your mental model a bit if you're coming from Jupyter. If I were doing something where rerunning cells wasn't expensive I would probably prefer the Livebook model.
[1] https://github.com/elixir-nx/axon/blob/main/notebooks/genera... [2] https://github.com/elixir-nx/axon/blob/main/notebooks/genera...
We have been working on this (Nx + Axon + Livebook + Bumblebee) for almost 2 years. One person full-time (part-time for the first year) and 3 part-time. But we have taken a foundational approach: instead of providing bindings to high-level libraries, we choose to build the whole foundation in Elixir and only leverage 3rd-party libraries at the compiler level. It looks like this:
|-----------|
| Bumblebee | pre-defined models
|-----------|
| Axon | NN library
|-----------|
| Nx | tensor library
|-----------|
| Compilers |
|-----------|
This design gives us agency and resiliency: we have control over how to evolve the whole stack and we can swap between different compilers depending how state of the art evolves. We are more expressive too, as we are not limited by design decisions done in libraries written in a separate language.
But you can skip ahead provide direct bindings to libraries such as torch + torch.nn. This will cut corners but also will officially tie you to a Python library.
However it is important to notice that embedding a neural network model inside your Phoenix app, as done in the video, is only really practical in a language that can fully leverage concurrency and effectively run both CPU/IO workflows in the same OS process. I am not following the progress of concurrent Ruby but my rough understanding is that it is not there. So in practice you would need to deploy your machine learning models to a separate service anyway.
I hope this helps!
The advantage here is data being fed into the model can use frameworks like Broadway (streaming ingestion) or Membrane (realtime media streaming), or Nerves (BEAM at edge / IoT), or as a controller for unreliable agents.
So performance shouldn't be an issue at all, when compared to similar "high-level language calling low-level language for heavy tasks" situations, like Tensorflow or PyTorch. It does, however, come at the cost of sacrificing the extreme stability BEAM applications typically enjoy: misbehavior by the backend can cause hangs or even crashes of the BEAM node they're running in.
Rust is an exciting option for NIF, but those backends are not written in it.
Few quick links:
https://www.erlang.org/doc/man/scheduler.html
https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...
https://bgmarx.com/2018/08/15/using-dirty-schedulers-with-ru...