josevalim

Another difference between the two comes from Ruby having open classes, so there is no callback that says "no further changes will be made to this class". This means you need to postpone some meta-programming to certain events, like Rails telling the app has done initializing, or until something is invoked for the first time.

So you are right there are distinct phases but they are established by convention and practices. And sometimes it is different between dev and prod (lazy loading vs eager loading). Or at least it was back then. :)

> I'm guessing nothing quite like this could exist for Ecto

Correct. :)

For the first you can setup a tracer for messages sent and received. Second should not happen from OTP 24.

Fourth is the wrong question to ask: if the failure handling mechanism is invoked too frequently, fix the failures, not the recovery.

Third is definitely my fault. Bug reports are appreciated whenever possible.

But generally speaking, yes, debugging is part of every programming language, and concurrency and distribution will add to the adventure whenever used, to varying degrees depending on the programming language.

Tesla is definitely fine to use! In my opinion it relies too much on module-based composition/configuration. In this aspect Req is more functional and less setup-y.

I believe `req` borrows inspiration from `requests` and applies a functional approach. Maybe it will become a similar standard in the community? The idea then is that you have the high-level library but the building blocks are available for low-level control.

You have a good point about HTTP though and I am not sure if it will improve given 1.1, 2.0, and 3.0 all imply different trade-offs as well.

To add to your excellent points: Elixir is compiled ahead of time while Ruby bytecode is generated as the code is loaded, which is probably the source of confusion in this case.

This leads to differences such as Ruby meta-programming happening at runtime, Elixir’s at compile time. The Elixir compiler (the Erlang compiler really) can also afford to do more work at compile-time which then leads to different approaches at the JIT level too.

When you receive, we will use the pattern and guards from the "receive" to lift type information, but we won't verify the messages between processes in our initial efforts. Personally, as future work, I am more interested in typing behaviours, because that's how most users define and interact with processes.

Folks tend to understate the influence of Erlang and overstate the influence of Ruby on Elixir. In a nutshell: Elixir is dynamic because Erlang is dynamic. :)

Ruby did influence the syntax and the names in the standard library, but the latter was also done in a more "democratic" fashion: I would look into Ruby, JavaScript, Clojure, Haskell, and choose a name that was common and closer reflected the semantics that would fit Elixir (for example, the term "protocol" come from Clojure).

The most curious case is the "Regex" module (which is named Regexp in Ruby). I quickly noticed that both "regex" and "regexp" were very common, Erlang simply used "re", so I had no clear direction to lean towards. The decision came by getting the top 50 languages at the time and using Rosetta Code to pick the most common term (which was regex).

> It's been a while but I thought the point was to experiment with static typing of XML transformations, but ultimately type inference complexity results hit a wall

Type inference for set theoretic type systems is expensive and that has not changed. It is not surprising either: inference is harder or undecidable in more expressive type systems.

We address this by providing reconstruction based only on patterns, guards, and return types. Other than that, inference is not a major limitation to us, given you don’t need to declare types today anyway.

The type system _may_ allow the type of a field to be based on the value of another field. I will investigate and, if possible, we would have to figure out a syntax that makes this possible to express.

The schema struct itself has all keys as required (the keys are always present). The map you receive on your changeset has all keys as optional because they can be missing (which we convert into a changeset error if the field is validated as required).

For the paper, we have decided to go as precise as possible, since its goal is to discuss the type-system semantics and not its syntax, so parentheses are everywhere.

You will for sure be able to drop the outer parens in your case and you _may_ be able to drop them on the function types too (to be decided). Ending up with something like this:

    $ (integer -> integer) and (true -> false) and (false -> true) and (a -> a) when a: not(integer or boolean)
    def negate(arg)

We will also allow intersections to be broken across multiple declarations:

    $ integer -> integer
    $ true -> false
    $ false -> true
    $ a -> a when a: not(integer or boolean)
    def negate(arg)

Initially we will only leverage information from pattern matching and guards. There are no plans to leverage typespecs for now (as someone said below, they likely won't be as precise).

And yes, the semantics should map 100% cleanly to Erlang.

Exactly. It was very important for us to figure out how well Elixir fits the type system, otherwise we would either get false positives or we would end-up relying too much on gradual typing. The other effort was to make sure we can lift as much information from pattern matching and guards.

We were 100% not expecting this to show up on HN front page. :)

In any case, the title is accurate in that it is only a prototype and it was built by Guillaume to explore how Elixir idioms map to set-theoretic types. The UX and syntax do not reflect how the type system would look in Elixir itself (if we get a type system at all).

You can find more context in Guillaume’s talk: https://m.youtube.com/watch?v=gJJH7a2J9O8

The initial research announcement: https://elixir-lang.org/blog/2022/10/05/my-future-with-elixi...

And the paper as an accurate and in-depth resource: https://www.irif.fr/_media/users/gduboc/elixir-types.pdf

This is a very nice resource, thanks for sharing!

No worries!

For what is worth, I was not considering Dialyzer as part of my reply. You should be getting many warnings related to typos from the compiler!

Anyway I agree with the sentiment: we have a high ceiling but we are currently far away from it. If we had a Dialyzer that runs all the time, is fast (Erlang/OTP 26 already improved here), and has good error messages, it would already be great.

We are currently researching a proper type system into the language. Fully integrating and improving the Dialyzer experience would be our plan B.

Those are two distinct capabilities.

Static analysis in Elixir is quite more accessible because you rely way less frequently on dynamic dispatch than Python (or Ruby or JS). You typically know which module you are calling to, structs and pattern matching tell you about fields (which we verify at compile-time) and primitive type information. Things like deprecated and undefined functions are part of the compiler, while most other dynamic languages typically require type systems or linters (or do it exclusively at runtime).

I agree on type analysis though. It is better than nothing but that's not much to talk about. We are working on it. :)

Hello, creator of Elixir here (and co-creator of Nx).

I like your breakdown of the requirements in three abilities. I think Elixir provides all three of them (with a caveat) and I will refer to two videos for those who want to learn more:

* This video shows how you can use Livebook (open-source computational notebook environment) to explore pre-trained neural networks (from Hugging Face), learn how they work, and then deploy them as part of a Phoenix web application: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...

* This video shows how to scale a neural network serving, first to run it concurrently, and then have it run distributed over multiple CPUs and GPUs: https://news.livebook.dev/distributed2-machine-learning-note...

The caveat is about your second ability. The production scale models are, at the end of the day running native/cuda code, using the same backends as Python (Google XLA and LibTorch). What we do is that we compile your Elixir code into a graph which is then compiled to the CPU/GPU just-in time (in an approach similar to JAX). You can find more information here: https://github.com/elixir-nx/#why-elixir

2. You should be running multiple Selenium instances (or equivalent) in parallel even on your machine (unless you run out of memory or CPUs).

3. Exactly. This is not a problem in Elixir. If it takes 3 minutes to render a request, all other incoming requests will progress and multiplex accordingly across multiple CPUs and IO threads. This also has a direct impact on the latency point brought up earlier.

I dropped the ball on this but check the persistent term compiler for Absinthe Schemas. It should make a difference. :)

I'd say those points deserve a deeper look. Take concurrency for example. It is not only about scaling, it can actually affect every step from development to production:

1. Development is faster if your compiler (or code loader), tasks, and everything else is using all cores by default.

2. You get concurrent testing out-of-the-box that can multiplex on both CPU and IO resources (important given how frequently folks complain about slow suites).

3. The ability to leverage concurrency in production often means less operational complexity. For example, you say you can offload complex documents rendering to a background tool. In Elixir this isn't necessarily a concern because there are no worries about "blocking the main thread". If you compare Phoenix Channels with Action Cable, in Action Cable you must avoid blocking the channel, so incoming messages are pushed to background workers which then pick it up and broadcast. This adds indirection and operational complexity. In Phoenix you just do the work from the channel. Even if you have a small app, it is fewer pieces and less to keep in your head.

At the end of the day, you may still think the bullet points from the previous reply are not sufficient, but I think they are worth digging a bit deeper (although I'm obviously biased). :)

> I mainly work in my company's monolithic codebase with about 800k lines of code, and compilation with even a simple change can sometimes take a full minute. A full re-compilation including dependencies is like 20 minutes. It's a bit much.

Agreed. My focus on Elixir v1.15 (rc to be released this week) was to improve the compilation time and I have seen increases of up to 30% in compile and boot times. It requires Erlang/OTP 26 as well. Once you upgrade, I would love to hear if it did change anything for you.

Hello, Elixir creator here.

Indeed! Almost everything that Erlang/Elixir (I'll mention only Erlang from now on) offers can be achieved elsewhere - but Erlang offers it all in a unified and cohesive package.

In Erlang, all of your code runs inside cheap lightweight threads of execution called "processes". They are all isolated from each other and run at the same, which brings us concurrency. When they need to coordinate, they do so by exchanging messages with each other. Message passing is location transparent too: you can transparently communicate between processes in the same or in different machines.

Can you build distributed software in other languages? Yes, but it often requires bringing in additional frameworks and libraries. In Erlang, I do `spawn_node(othernode@localhost, fun() -> ... end)` to start a process in another node and that's it.

Can you scale in other languages? Definitely. But Erlang offers both vertical scalability (via processes) and horizontal one (via distribution) out of the box.

> In a run of the mill corporate Java app, if you experience an exception eg calling another service, unless you're doing something very strange, your app will not stop running

Correct. But this is achieved by having the web server or the framework wrap all user code into try/catch or similar constructs. It is a defensive style of programming where "forgetting a try/catch" somewhere leads the whole system to fail. In Erlang, everything run in isolated processes, so the _default_ mechanism is that crashes do not cascade.

---

You probably noticed that I wrote "processes" a lot - and that's the point. You can build scalable, distributed, and fault-tolerant systems in other languages, but it often requires picking specific solutions to each of those problems. A thread-pool/scheduler for certain concurrency patterns, a network library (or an external system for distributed communication), plus defensive coding. Erlang offers a single abstraction, called process, that encapsulates all of these concerns.

I often like to draw comparisons with k8s too. Certainly it was possible to orchestrate and deploy large systems before k8s but putting everything into a unified package does wonders to streamline the experience. I wrote comparisons between the two in the past too (if that's helpful): https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

I am not sure what we could do better on the marketing - but if you believe we could improve something based on the additional context in this thread, let me know.

PS: +1 to Sasa's talk which others have already mentioned.

Edit: I wrote down an introduction to all of these on our elixir-nx organization page on GitHub: https://github.com/elixir-nx

Languages and runtimes often grow beyond their original scope. And since the introduction of Dirty NIFs to the Erlang VM five years (or so) ago, integrating with native code (which is what powers a lot of data analysis and machine learning tools in high-level languages) has become a real possibility. There is a similar-ish discussion here: https://news.ycombinator.com/item?id=35572128

Things that are in our roadmap in relation to our data vision:

* Data management within your notebook (https://github.com/livebook-dev/livebook/issues/1604) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it

* We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by selecting a visualization: https://github.com/livebook-dev/livebook/issues/1545

We have other ideas, such as making SQL a more prominent citizen in Livebook and be able to build a custom canvas as your work on your notebook, but those will likely take longer to realize.

Hi, maybe I can try to answer your question.

First, to makes ure we are on the same page, distribution in Erlang happens across nodes and concurrency happens within a single operating system process. Erlang calls its concurrency primitive processes (because they are also isolated and preemptive) but that can cause some confusion (hence this comment).

From now on, when I mention process, I mean the Erlang VM processes, and they are very lightweight and you can create millions of them.

I can think of a few different ways where concurrency can help interactive data analysis:

1. Livebook supports rich outputs where each output is a process. This means your notebook can communicate with outputs as it executes. For example, it is very easy for you train a neural network and push data to the graph as it comes. Or to process data and plot it as you go.

2. You can use concurrency to run several experiments at once within the same notebook. We support this in Livebook via "Branched sections". You can prepare the data and then start several branches/processes to digest the data in different ways without a need to start several notebooks.

3. It is also very easy to build applications where multiple users can collaborate and interact with it, which we showed yesterday: https://news.livebook.dev/build-and-deploy-a-whisper-chat-ap...

When it comes to distribution, it is quite similar to above, because the concurrency and distribution primitives in the Erlang VM are the same. Here is an example of how easy it is to take a ML model from concurrent to distributed: https://news.livebook.dev/distributed2-machine-learning-note...

Generally speaking, I think we should start from the opposite side: we should try to make everything concurrent by default and fallback to serial only when we cannot. Specially for data analysis, where moving data is expensive, we may end-up incurring a lot of overhead if the only form of concurrency is via the network or inter-process communication.

One last note, perhaps the most important bit of the Erlang VM machine for data analysis is that it favors a functional style. Livebook notebooks are strongly reproducible. I expand on this in this video: https://www.youtube.com/watch?v=EhSNXWkji6o

I hope this helps (and feel free to tell me if I missed the mark!).

PS: I know many of the videos above are machine learning related and that's because we have started our data journey only now. Although the principles should generally apply. Hopefully more data videos will come soon! :)

José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

* Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

* Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

* Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

I hope this helps!

Hi everyone!

I am back for day 4 of Livebook's Launch Week. This time we use Livebook to build a Whisper chat app - where you can only talk via audio transcriptions - and deploy it to Hugging Face. All done with Elixir!

You can give the deployed application a try here: https://huggingface.co/spaces/josevalim/livebook - and then click on "Whisper chat" on the right. Audio messages are concurrently handled and batched within the same instance before being handled by the GPU. I guess we will soon find out if a Nvidia T4 is enough. :)

Have fun and I am glad to answer questions once more.

The documentation for distributed servings can be found here: https://hexdocs.pm/nx/Nx.Serving.html

Or are you looking for something else?