josevalim

> These libraries just aren’t “there” yet, and I think it’s important to be honest about that.

Exactly. These projects are still during early adoption stage.

While it is production-ready and many are doing so, getting there requires improving documentation, discussing roadmaps with maintainers about a feature you may need (perhaps even contributing it), giving presentations (at your company or at events), and so forth. This can be exciting to some, but a distraction for others. There is certainly plenty of work ahead.

> People have to realise these are mostly for hobby. It is really hard to get these working with other libraries.

I have been dealing with this type of feedback for almost 20 years at this point: "This library doesn't work for my needs, so the conclusion is that it is hobby/not useful/etc for everyone". In your experience, we are not there yet (and we may never be), but the generalization does not hold.

Take Explorer, it was the first library of the bunch to be used in production because it was created by Chris Grainger to write data pipelines at his company. And the blog post does include the link to a talk on production usage.

Other projects, like Nx, started as an experiment, but nowadays it is used in production quite a bit and we spend a lot of the time refining it based on feedback we receive.

No one is claiming those libraries are complete. It is still pre-1.0. But we claim to be evolving and improving based on feedback. So if anyone wants to try something different than Python, and you can afford to engage, give feedback to maintainers, and even contribute, then you will hopefully have a good time and be closer to running it in production too.

You are not wrong but you may be hinting at the wrong cause. It is hard to generate a useful graph or summary that gives actionable feedback once you reach a certain amount of nodes. Even mix app.tree, which has no cycles, is barely useful after 30-50 deps because it becomes dense.

Using mix xref to evaluate your project will, at best, give you a very blurry picture. Think of it more of a database: it will give you good information when you need to answer certain queries.

I am Mox author and, honestly, I rarely use it. The most important part is to think strongly about what you are mocking and your interfaces (explicit or implicit). Once I do that, I do my best to use plain anonymous functions, modules, or protocols to test it. If you already have this discipline, then Mox only adds “paperwork”.

If you have layers of interfaces, it is worth asking why you need mocking across several layers.

YMMV, of course. As long as you are mocking with care, use whatever floats your boat. :)

I enjoyed reading the previous articles, so I was excited to read this one too. I really appreciate this style of feedback. Comments below.

-------

ERROR HANDLING

In my opinion, the "with {:is_email, true}" style is missing the forest for the trees. The whole point of "with" is to match on a consistent result. If you need to distinguish individual clauses, then you should either use case/cond or normalize the result types, in the same way you would do in Rust. So in your case I'd add two functions: "validate_email" and "check_email_availability" that returns ":ok" or "{:error, reason}". Then you end-up with:

    with :ok <- validate_email(email),
         :ok <- check_email_availability(email) do
      ...
    end

We give similar examples in our anti-patterns docs: https://hexdocs.pm/elixir/main/code-anti-patterns.html#compl...

-------

STATE MANAGEMENT

Generally agreed. Just one nit:

> Sure the state is all encapsulated into processes, but then those processes are hidden behind an abstraction layer that makes them invisible, so really you’re just touching global variables.

They are not invisible. You can use Observer, the Phoenix Live Dashboard, and many other tools to traverse, explore, and navigate the supervision tree, processes, and see where the state is!

-------

IMPORTS

Agreed. We had several discussions on how to improve this but nothing satisfactory. Maybe it is time for another tango.

-------

MIXED MESSAGES

I'd say we actually do a good job on the official docs on the topics that are directly related to Elixir:

* On umbrella projects, the official guide discusses trade-offs: https://hexdocs.pm/elixir/dependencies-and-umbrella-projects...

* Live upgrades are covered in our release docs: https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-cod...

* On macros: https://hexdocs.pm/elixir/macro-anti-patterns.html#unnecessa...

The trouble is in finding this information, as it can be a lot to absorb. If anyone finds we should link to them from other places, pull requests are welcome. In general, PRs to improve docs are always gladly received, be in Elixir, Ecto, or elsewhere!

-------

OTHERS

> Anecdotally, when Elixir started off there was some bad blood between them and the Erlang community, which is the origin of this schism

No bad blood, really. I asked the Rebar team (not the current Rebar3 team) if they would accept PRs to also compile Elixir, they said no (which is understandable) and then we move forward with Mix (which was a contribution from a Clojure developer inspired by Lein). The projects drifted apart but we often share whatever we can in other places (such as https://github.com/hexpm/hex_core).

> In fact the Elixir compiler almost never gives you an outright error, basically it only fails if a file can’t be parsed. This feels spooky as hell… but its warnings are basically always correct and seldom miss anything

Yes! Our goal is to avoid halting compilation as much as possible and instead rely on precise warnings. It is easier to debug a program that compiles (and then raises) than one that does not compile at all.

If you ever get to what is bothering you on unit tests, I'd love to hear (feel free to reach out).

Exactly! The hard part was to find a type system that we can map most (if not all) Elixir constructs to, and set-theoretic types have been a great fit so far and we were able to develop the theory in the areas where it lacked.

It is not a goal of the type system at the moment, as we will focus on software correctness and developer experience.

However, keep in mind the Erlang VM already uses typing information (from patterns, guards, and built-in functions) to optimize code through its JIT compiler.

The examples will be very gradual - no pun intended - as we need to go through each Elixir construct and either add inference (if a pattern or a guard) or type checking rules. Right now we infer and type check one single construct (bitstring) but the foundation is there to build on.

Thank you for the nice words, it is appreciated and it makes a difference whenever I am having a bad day. :)

There is a similar discussion here: https://news.ycombinator.com/item?id=38544486

TL;DR: you get granular and programmatic scaling, including the ability to scale using specific resources (for example, machines with GPUs for certain workflows).

Yes, the local backend runs on your machine, and you would debug it like any other Elixir code on your project/machine.

Wouldn’t you lose, for example, streaming capabilities once you use Celery? You would have to first upload the whole video, then enqueue the job, and then figure out a mechanism to send the thumbnails back to that client, while with FLAME you get a better user experience by streaming thumbnails as soon as the upload starts.

I believe the main point though is that background workers and FLAME are orthogonal concepts. You can use FLAME for autoscaling, you can use Celery for durability, and you could use Celery with FLAME to autoscale your background workers based on queue size. So being able to use these components individually will enable different patterns and use cases.

I believe the reason why we have seen so many different numbers given as limits for distribution is because they are often measuring different things. As you know, the limits will vary depending if it is just distribution or if global/mnesia are used. I have seen distribution by itself going north of 300 nodes (and heard from others going quite beyond that) without issues and heard reports of global usage struggling past 40-60 nodes (especially global transactions).

At least in the Elixir ecosystem, global is rarely used and I personally suggest looking for alternatives if you are going past 20 nodes. Unfortunately I have no practical experience with mnesia.

> Relying on shapes instead of types to identify results / objects

Is the issue lack of types or relying on shapes? Those can be orthogonal features, since you can have structural typing (i.e. types that are built on top of shapes). Nominal typing is often too rigid (and not the direction we are exploring in Elixir's type system).

> for example behaviors / protocols should be one implementation instead of two separate ideas

Elixir has three building blocks: processes, modules, and data. Message-passing is polymorphism for processes, behaviour is polymorphism for modules, and protocol is polymorphism for data.

The overlapping comes that modules in Elixir are first-class (so they can be used as data) and that data (structs) must be defined inside modules. So while maybe those features could feel closer to each other, I don't think they could be one implementation. One idea would be to treat modules/behaviors as "nullary protocols" but that's not what "nullary typeclasses" mean in Haskell, so it is quite debatable.

Do you have examples of languages that do both in one implementation? For example, my understanding is that both Haskell and Go offer "protocols" but not "behaviours". I'm not aware of a language with both as one.

I'd love to hear other concepts that overlap, it is always good food for thought!

It runs fine, but using metal is complicated. We provide bindings for both libtorch and XLA. Libtorch has better metal support but still incomplete, with types and functions missing. Apple released a driver for XLA and it is, unfortunately, closed source and tied to a particular XLA version, so I wouldn’t particularly recommend it.

Hi everyone, author here. This is a roundup from multiple projects and collaborators within the Elixir+ML ecosystem. I will be glad to answer questions where I can. :)

Sorry to hear this. Phoenix v1.7 changed how it structures files in disk and that broke quite some of the getting started material. However, the guides are always kept up to date, so you can give it a try: https://hexdocs.pm/phoenix/overview.html

You can also see the resources on this page listed by year: https://github.com/phoenixframework/phoenix/blob/main/guides... - the recent launched ones are most likely up to date.

Hi HerrBertling! I am glad you are enjoying Elixir and learning new things, I appreciate the nice words and the opportunity to be a positive influence. :)

> Erlang and Haskell have been hovering at under 1% for several decades.

I am well aware that I am nitpicking at this point, but putting a restriction on your metric that rules out 99% of programming languages makes the comparison quite limiting. :D

You are ultimately only speaking about languages that are already quite popular, because even languages like Kotlin, Dart, and Swift (all endorsed by billion dollar companies) are hovering around zero by the metric you provided, and I would have a hard time classifying them as a noise. And languages like Go and SQL are still well below the 5%-10% reference point.

Hi jerf! I feel this is evaluating Erlang from an angle it is not really prescribed as a solution for.

Erlang's goal was never to be your company's message bus. Or your k8s. Even hot restarts have always been advertised as an extremely specialized feature, justified only for very specific use cases. So you are right, if someone wants to use Erlang for these specific purposes, then I would ask them to triple check their decision.

> Other features, while nifty, like process restarts with new code, just turn out to be more niche than the brochure may advertise. That's a super important feature for phone switches, yes, absolutely, but it's never actually been something I need

Agree/disagree. I agree that fault-tolerance must not be an application developer concern. However, it is a huge plus to know that my web server, connection pool, ML serving, and other essential libraries are backed by supervisors instead of crashing and getting into a unknown state. And those are errors a type system won't catch anyway: "oops, your database sent a tcp packet you were not expecting, good luck!"

> the integration itself is a feature that very few other platforms can compare with! While I can find every feature now in other languages and environments, I will be putting together a stack of my own somehow, which can be beneficial, but is also definitely a cost of its own too.

Agreed and that's precisely the selling point. If you have to build an application that needs to concurrently talk to several APIs, or perform background processing, or do real-time messaging, or distribute machine learning workflows, then you can build such application using Erlang/Elixir without bringing Kafka, Redis, or much else into the equation. Replicating this elsewhere will often require stitching three or more technologies together, which become additional techs to learn, master, and maintain on every step of development, test, and production.

In any case, I still find it "incorrect" (sorry, I can't think of a better word) to judge Erlang and Elixir by the unmet potential of its _exceptional_ features. Even if you strip distribution, hot code swapping, supervisors, they are both excellent dynamic languages with concurrency capabilities that are not seen on other mainstream dynamic languages (although I am obviously biased). If you are chasing system-level performance or static-typing though, then they are obviously not a good fit.

Ultimately, I find the question of adoption is way less technical and rational than most people expect to. That's something we often hear about technology in general, and I don't think programming languages, frameworks, etc, are much of an exception. :)

> One thing I would add, is Erlang/Elixir's emphasis on multi machine concurrency is somewhat at odds with the serverless future we're moving towards

IMO, that's a misconception. They work at different levels and I wrote a more in-depth explanation here: https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...

Although I agree the sentiment plays a role on adoption, because many ask "why Elixir when I have k8s", while in practice, if you like k8s, you should enjoy Erlang/Elixir. They apply similar principles at different scales.

So I don't think we could have made a dent on k8s, but rather, we should have positioned ourselves earlier within k8s strengths. :)

> it is rare that a language doubles its market share (assuming it's not in the vicinity of zero) after its first decade

JavaScript, Python, and Ruby are all counter-examples. Erlang and Haskell might as well.

I don't think time plays such an important factor. A language may exist for 20 years and then a large corporation or an all encompassing framework draws attention to it, leading to unforeseen growth.

> In other words, if you're not at 5% market share by age 10, chances are low you'll ever reach 10%.

This won't happen for 99% of programming languages and it shouldn't be a goal or even a comparison point. For example, on GitHub language stats [1], only three languages have more than 10% activity, only 7 within 5% activity. Pick other rankings, market share related or not, and you will find similar numbers. Tiobe lists SQL, Go, and PHP all below 2% and I don't think anyone could argue they failed at market success.

> So getting to 1% and generating hype is not as hard as showing stable and growth and getting a large community that sticks with you for over a decade.

Given all of the above, I strongly disagree. Getting to 1% is incredibly hard.

[1]: https://madnight.github.io/githut/

If it comes to this, we will certainly need to have a discussion on the topic. So I am definitely open to discussing the idea!

It is impossible to ignore the set theoretic type aspects, because the whole foundation of strong arrows is based on being able to negate a type. If I don't explain the set theory bits, then the rest of the article falls apart.

I could perhaps rearrange some of those sections but that doesn't change my main argument: the type system is not being designed for type theoretic people. At best, you could say the _article_ focus too much on theory, rather than on the practical bits.

This is a surprising assessment given most of the blog post is explaining how to build the type system in a way it can give feedback on all _existing real projects_ already written in Elixir. :)

It is the opposite of optimizing for type theorists and rather exploring how to bring values to the existing millions of lines of code.

Edit: I am the author.

I understand many would love to have this yesterday but I strongly believe we should not speed it up. :)

This is potentially the largest change the language will ever go through and we are being very intentional and deliberate on every step and decision we make. Once we start collecting feedback from the community, it may speed up or slow down the process, but I want to make sure everyone gets plenty of time to experience and internalize the changes.

When it comes to funding, the on-going work on Elixir type system is well funded by companies like Fresha, Starfish*, Supabase, and Dashbit. However, we also want to venture into new research areas, such as existential set-theoretic types, which would allow us to type behaviours (such as GenServer) similar to how one would type first-class modules in OCaml (or Typeclasses in Haskell). If someone is interested in sponsoring research work, hit me up (github.com/josevalim), and I can put you in touch with the proper institutions.

> In other words, is the "set theoretic type system" an implementation detail which doesn't affect the language server API?

This should be the case. Type violations would automatically appear on your IDEs when you update Elixir. However, if you need additional metadata/features, then we need to expose them from Elixir and Language Servers need to consume it. But that's implementation work and not "type system" work. :)

> But regardless, this is one of my main hangups with dialyzer: its `underspecs` and `overspecs` design gets it wrong.

I believe recent Erlang/OTP versions changed this to give you more fine grained control for the precise reasons you mentioned.

> So I guess my question with this approach, is whether you can explicitly spec "larger" sets than is inferred? And if so, whether the given specs overrule the inferred ones elsewhere.

I am 99% sure the theory allows both options, so we will pick one based on patterns and community needs.

> Unless I didn't understand correctly, surely 'none()' is also a subset of the codomain?

Correct. Both are included for clarity.

> But does it also mean you can 'weaken' an arrow if you implement a more general version of a function?

If you have a function that is a strong arrow because of other functions (i.e. the current function does not explicit guard its arguments) then that would be the case.

However, we don't know yet how much of a big deal this is. Strong arrows is all about making static typing more useful to dynamic code. Dynamic code should not be relying on strong arrows for correctness, they are a nice to have. If you want guarantees, then you need static typing.

It may be that, in practice, it is enough for Elixir standard library to be made of strong arrows and that will propagate enough type information to most dynamic programs. Otherwise, if more is needed, we will definitely need to add visibility and reflection APIs around strong arrows.

You will find examples from companies moving away from Elixir, Python, JavaScript, Scala, Java, etc. I don't think there is any programming language that is "unditchable". So I'd suggest to analyze the reasons that made a company move away from X, check if they are still relevant, if they'd apply to you, and, if yes, if they are a deal breaker or something you can work around.

For the Brex case, if I remember correctly, lack of static typing and early-development of gRPC tools were the reasons back then. How much this impacts your needs is up to you.

> I keep hearing everyone talk about how Erlang/Elixir gives everything out of the box and you don't need to worry about Queues, RPC or whatever...

Many companies are using Distributed Erlang but not the way you described: they are using it to exchange messages between nodes running the same version of the software. Imagine that you are building a web application, you can directly exchange live messages between nodes without Redis/RabbitMQ [1]. If you are deploying a machine learning model, you can route requests through multiple nodes and GPUs without additional deps [2]. If you want to see which users are connected to your app, you can exchange this information directly between nodes [3].

In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

If you need to communicate between different systems or you need to persist data, then I 100% agree with you, I would not use Distributed Erlang (at best it would be one part of the solution). I think this distinction gets lost in many conversations and sometimes it leads to false dichotomies: "why Erlang when I have k8s/grpc/redis?" while in practice there is not a lot of overlap. I have written about Erlang/Elixir vs Redis [4] and vs k8s [5] for those interested.

[1]: https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html [2]: https://news.livebook.dev/distributed2-machine-learning-note... [3]: https://hexdocs.pm/phoenix/Phoenix.Presence.html

[4]: https://dashbit.co/blog/you-may-not-need-redis-with-elixir [5]: https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...