Unpacking Elixir: Resilience | Hacker News Reader

Unpacking Elixir: Resilience

underjord.io

213 pointslawik57 comments

A very nice description of Erlang/Elixir resilience.

> Fundamentally, if there is nothing useful your code can do on a failure, no mitigation, no meaningful fallback, then you might as well have it blow up and rely on the underlying tree to make whatever is most useful out of it.

I would add this is possible only because process heaps are isolated. Other systems with traditional threads/co-routines/green threads can mimic supervision trees, but unless they have isolated heaps and can safely crash at any time never affecting any other processes, it would be hard to achieve the same safety properties.

> Scheduling and preempting was intended to produce consistently low latencies but it also protects you from heavy workloads bogging things down, it prevents infinitely looping bugs from slowing the system to a crawl and in general makes things resilient to things not being ideal all the time.

Well put. Reliability was one of the initial requirements of Erlang, but soft real-time properties was another one. This is hard to do, but BEAM VM does a great job there. To get this right the VM has to be able to preempt processes, even if they run a tight CPU bound loop. A misbehaving process may not just crash, it could just endlessly recurse, burning CPU cycles, but still shouldn't bring the system down.

> The strategies are normally one_for_one, one_for_all and one_for_rest

Minor correction, the last one is rest_for_one https://www.erlang.org/doc/man/supervisor.html#supervision-p...

One of the great things about the Elixir ecosystem is that, in so many cases, the libraries you’re using make the appropriate use of supervision trees so you don’t have to.

Obviously it depends on what you’re building and you can craft your own supervision structures if you want to. But most of the time you can get the benefits of BEAM resiliency without having to drop down into the details.

The one thing I dislike about Elixir is the confusion caused by how some libraries execute their functionality in-process and others spawn a supervision tree and message pass - putting your work in a possible queue that can get overloaded.

Its a common pitfall for more inexperienced devs and I think its just a symptom of the language conventions.

Everyone is taught genservers with a client-side API existing inside the same module as handle_ functions and it blurs the distiction some.

I think if the convention was to create a separate module to implement the client-side convenience functions then the client-server split would be more obvious.

> The one thing I dislike about Elixir is the confusion caused by how some libraries execute their functionality in-process and others spawn a supervision tree and message pass.

It's my assumption that needless process-starting is from classic OO developers trying to get something closer to the objects they're used to. I'm not sure separating the client and API logic in tutorials would necessarily stop them from doing this. People who are hellbent on shoehorning familiar idioms into an unfamiliar language are going to do so regardless. But I see what you're saying.

I think it's more important to drill home that modules and processes have nothing to do with each other. I still catch myself conflating them once in a while three years in and it's caused the most confusion while trying to teach friends (although I'm not a particularly good teacher).

Yeah, there are a few common mistakes that can get really troublesome when library authors make them. Assuming how you want to structure execution (by using GenServers in the library usually). Another one is only one instance of config. Meaning if you have a plan to use the library for multiple distinct activities or with multiple accounts, you have trouble.

Both simple config and GenServers have their place in libraries. But if a library doesn't expose the fundamentals it gets gnarly quickly.

This is an allowed practice, just not idiomatic, mostly following on Erlang convention I think. If you do the coding gnome elixir course, he quite explicitly has a section on this and proceeds with separating API from server. At the end of the day, it is equivalent and all it takes is getting used to one or the other. I would argue spending the time to get used to the idiomatic way a community establishes is to your benefit in the end.

I agree and love Elixir and use it daily. I think my downvotes above is really strange lmao. This is just a common footgun I see new developers struggle with.

My guess re: downvotes is the wording. The opening sort of reads like you have written-off Elixir entirely because of this one issue. That was the impression I got, at least, though I was able to figure out that wasn't the case before I responded.

I feel like Elixir get pushed too hard on people. Makes me skeptical. Same way I feel about TypeScript community. Calm down, it's just a programming language.

I don't believe the programming language matters much. What matters is one's familiarity with it. There are good tools and frameworks available for just about any moderately popular language. Every language has caveats. Every language has lots of pros and cons and when you average them out, they're essentially all the same.

Just a matter of personal preferences and habits.

It's not just a programming language—it's an entire philosophy centered around fault tolerance and concurrency, thanks to its foundation on the Erlang VM. While Elixir's syntax and features are certainly appealing, the real power lies in its integration with the Erlang VM. When people praise Elixir, a significant portion of that praise is directed at the capabilities you get with OTP/BEAM right out of the box. These include robustness, distribution, and real-time processing, which are not as readily available or as mature in other ecosystems. the Erlang VM is 38+ years old.

It's an Erlang convention. Erlang suffers from having a very rudimentary module system, so a clean separation between the public API and the gen_server API isn't that easy to make. If you have access to a better module system, you should definitely induce a cleaner separation and adapt that as a convention. It helps people in general, but the in particular the less experienced.

When you say 'execute their functionality in-process', what exactly do you mean? In the context of the current process? Are there any good resources explaining when it is more appropriate to do one over the other?

One example is HTTP libraries.

For instance, take Mint (https://github.com/elixir-mint/mint):

> Mint is different from most Erlang and Elixir HTTP clients because it provides a process-less architecture.

Mint is a low-level library which doesn't make attempt to manage processes (including HTTP pooling).

In contrast, Finch (which builds on top of Mint) includes pool management:

https://github.com/elixir-mint/mint#connection-management-an...

It can take someone a bit off guard when they realise that the library they use provide a "default pool" they were not aware of, and that it can become a bottleneck etc.

Great, thanks for that I'll take a look.

There's a library for embedding inline svg icons. You call it with a string name in your html template and get back a string svg.

It was originally implemented as a single GenServer. When you app started up it'd read the filesystem and create a map of svg name => svg contents in a process. Then each lookup call would send a message to that process and get back the contents.

This has had the flaw of making template rendering single threaded. It's far worse than a file read per lookup

This sort-of stems from the underlying OTP principles from Erlang. Each application has their own isolated supervision tree, and your system contains a number of applications.

Hence, a library which contains a service is likely to have their own internal supervision tree. Your code can then sort-of latch onto this tree and piggyback on its existence. This is very common in web applications because when a new connection is opened, you spawn a process to handle the connection. If connections are (de-)multiplexed into channels like in HTTP/2 and HTTP/3, you'd just move up one layer and spawn a process per channel.

Likewise, if you have a limited resource such as Postgres connections, they are usually maintained in a pool, and access is queued until a connection from the pool is ready for use. This can circumvent the need for something like pg_bouncer in some use-cases.

The thing about Elixir resilience is that it takes experience to fully appreciate what it does and realize that in a web app, the actual use case when this becomes important is not that often to be totally honest.

I can give an example where this could be useful, I once worked on a node-app that couldn't use clustering mode because reasons and we only had 1 server running this particular piece of code. It turned out that we had a bug that made the node app crash whenever a user posted some invalid data. The problem here was that the app itself was a SPA and the users could just keep on posting even if it failed and they did in frustration.

So then app crashed and took a few seconds to reload, then crashed again. This mean that the entire api went down while the user was posting and thus could not respond to any other requests. This would never happen in Elixir and the load would just continue being ok even if 100 users at the same time would keep posting bad data.

The bad thing about Elixir resilience is that it is only applied to application logic. The rest of the time shit can go wrong is just the same as any other app since most elixir projects use the same kind of tooling (postgres, some web server in front etc). Not that many seems to use the built in mnesia database, no downtime deployment etc. The BEAM comes with many cool feature in theory but very few actually utilizes them so this 99.99999% uptime rarely comes into effect. The amount of time I've had downtime on apps because of things in the application logic like the first story I mentioned has been very, very few and most of the time it's something else entirely and that thing does Elixir not really help with most of the time.

Sure you could utilize all the cool features of the BEAM but it seems like in the absolute majority of cases the amount of work is simply too great for it to be worth the time investment required.

Nice article! FTA: "One for one indicates that if a child crashes it, it and only it should be considered when restarting." What is "it" in this context?

The one_for_one policy means that if a child process terminates, it will get restarted. It is appropriate when the child processes managed by the supervisor are (relatively) independent. The contrasting policy is one_for_all where a terminating child will also induce a restart of all the other children of the supervisor. This is appropriate when there's a number of important dependencies between the children processes, and you'd better just rebuild the whole subtree.

It is key to run the thought process of system failure. You need to understand what will happen if a certain part of your system errors out in an unforeseen way and make sure your restart policies are likely to right the ship.

The should be a first comma after “crashes” IMHO.

“It” is the child.

So if the supervisor (“parent”) has other children, they won’t be restarted

What I also like about Elixir is, that it is a sort of nice language. What I mean by that is, that it has convenient language features like tail-call optimization, immutable by default data structures, modules (seemingly proper ones, but I am still learning, so I still need to check things like "Can I limit what a module exports?"), and great pattern matching. Then of course there is the heritage from building on top of BeamVM and Erlang: Lightweight processes. I also prefer having `do` and `end` markers for things, so that one can write expressions inline, like `fn args -> foobar end`. Having such markers enables structural code editing and things like putting the point at the `do` and then by one keyboard shortcut select everything until `end`.

Partially related question:

There was an article about the implementation of supervisors and you can’t use the “let is crash philosophy” in the whole erlang/beam stack but there is a small (loc) implementation in C which has to be proven safely in order for the “let it crash philosophy” to work.

I don’t know if this was the definition and implementation of a supervisor itself or a process itself.

Does anybody know what I mean and which article I am referring to? If so, I would be glad if you could post a link to the article

Elixir seems to be picking up insane steam right now. Every day or two there is a fascinating Elixir post here and its promise seems too good to resist. Has anyone else latched their cart onto this horse?

I built a fintech backend system using Eljxir a few years ago, and I loved it. However, I had to use Go and Python as a sort of escape hatch when companies offered massively time-saving SDKs in those languages but not Elixir. The ecosystem doesn’t seem to have really grown much since then. Looking around at the state of things, lots of important OSS packages have not been maintained, and that’s concerning. I absolutely love Elixir and the careful thought put into so much of the language and primary frameworks. However, I have become more skeptical that the actual ecosystem will hit critical mass. I really wish it would. I see a lot of articles on HN about Elixir, but my feeling is that it’s just more interesting than the constant stream of JS and Python we’ve had for decades now.

In my experience a lot of OSS elixir libraries are basically done. They get updated if the language changes, but otherwise they just do what they do well and they resist feature creep.

Elixir is one language where you can find a library on Github that has not been updated in 6 years but will work flawlessly.

I actually do small PRs to update deps etc on those kinds of repo and people are usually are OK to merge.

As someone active in the community I wouldn't say there has been any particularly massive ramp up recently.

Someone refernced ElixirConf US and some of the news-flow is certainly from that.

Elixir has been building a steady steam pressure for a long time and I think steady is still the case.

As someone writing about Elixir I will say that it has good traction here as a general rule. I think Erlang appeals to the CS-interested, Elixir appeals to a lot of Ruby and startup folks. The ecosystem also innovates a decent bit. I think it punches above its weight on the HN front page.

It is a healthy ecosystem and seems to grow.

(edit: a spelling)

The company I started working for a few months ago wrote their whole web app with Elixir/Phoenix LiveView. It's been a real pleasure to work with after years of React and Next.js. The main thing I miss are types though.

Are there no typed languages on BEAM?

Or some Typescript style front for Elixir?

BEAM is pretty deeply untyped due to its fundamentally untyped [1] communication protocol. There's some logical reasoning behind this choice, which is that by not having a typed communication protocol, it can defer handling mismatched types to the application layer. Since it is deeply designed with the knowledge that it will be used in a clustered context and that it is inevitable that there will be version mismatches within a cluster (during upgrade if nothing else since simultaneous anything is impossible within a cluster, including upgrades), this is a somewhat reasonable compromise approach to the problem. If you do add typing on top of BEAM you still need some sort of solution to this problem with will probably have some sort "unsafe" equivalent.

[1]: In the sense you mean here, anyhow. It has a selection of atomic types, int, list, tuple, etc., but there's no "user" types as one would have a strong typing system for. It's not an exact match but if you imagined a programming language that had nothing but JSON, not Javascript, Javascript has user types, literally just JSON as your underlying data model, you get pretty close to BEAM/Erlang.

Types in Elixir are on the roadmap [1] but I don't think there's concrete ETA

For now you have typespecs and dialyzer that tries to infer types and detect incorrect assumptions in code but I found it cumbersome to work with. Testing in Elixir is first class so I prefer that for now. You get doctests (documentation that is also a runnable test), like in Python, which I find very useful when looking at some package source.

[1] https://youtu.be/giYbq4HmfGA?si=BjNyOc5cjWWA7ER6

I’ve found that function head pattern matching and guards get you very close to the benefits of types.

https://gleam.run/ Gleam is super promising.

why would you use Gleam instead of Elixir, is there any benefit to using both of them?

The benefits would be if you prefer gleam:s syntax and static typing. In the end all compile down to the same Erlang byte code that runs on the BEAM so neither has any performance benefits.

ah ok, reading more compiles to JavaScript, generates typescript definitions, can reuse code from all Beam supported languages.

So maybe Gleam as an easier way to integrate JS and Erlang / Elixir?

Elixir gets a lot of praise but very rarely mention that is't a dynamically typed language, people will talk about Dialyzer/type hints but it's vastly inferior to strongly typed language.

I know people mean different things when they say this but Elixir is strongly typed in that it doesn't auto-cast. Furthermore, it goes farther than most languages in that it doesn't overload many operators. For example, string concatenation is done with `<>`, not `+`. This rules out many of those "subtle bugs" you would get in many other dynamic langues. For instance: `def compound_word(left, right), do: left <> right`. Even though there are no types specified (or even patterns of guards) this function can only every succeed with strings. Yes, it's still dynamic and will only be caught at runtime, but there are things about Elixir that make it a really nice dynamic language.

Python gets a lot of praise too despite being a dynamically typed language. People will talk about MyPy/type hints but it vastly inferior to statically typed language.

(Elixir is strongly typed btw)

Elixir is essentially the functional python.

https://elixir-lang.org/getting-started/typespecs-and-behavi...

"Elixir is a dynamically typed language"

Jose Valim is investigating set-theoretic types, as you may have heard: https://news.ycombinator.com/item?id=37593967

We've been running a large web application backend written in Elixir for over three years now. It's awesome and we've also rewritten some of our microservices to utilize Distributed Erlang. No need for a service mesh this way. :-)

We're using on the French national access point to transportation data (http://transport.data.gouv.fr/?locale=en, source code at https://github.com/etalab/transport-site) and we're not going to switch to another stack ^_^.

Plenty of use-cases are made easy by the stack, from regular web work, API (https://transport.data.gouv.fr/swaggerui) to proxying (https://github.com/etalab/transport-site/tree/master/apps/un...) to real-time maps (https://transport.data.gouv.fr/explore), clustered maps (https://transport.data.gouv.fr/explore/gtfs-stops), XML queries building (https://transport.data.gouv.fr/tools/siri-querier?endpoint_u...)...

The maintenance story is very good too.

With ML being added (Nx/Axon etc), and mobile apps being in the works (LiveViewNative), it has become my everyday language & stack.

Part of this is that ElixirConf 2023 just finished up, so we’re getting the news output of the talks given and libraries released.

Seeing a lot of rust , elixir, JavaScript but not seeing much clojure . Wonder how clojure compares to elixir in terms of developer productivity

Been running our tech stack on Elixir for the last 3 years. I'm the owner of a small e-commerce company where I'm the only tech person (and also the ceo so I can only spend 20% of my time on tech).

Why I love it:

- The mental model just clicks for me. The syntax is really simple and the semantics are consistent. There is no abstraction. It's all about processing data.

- The REPL (IEX) is way more than a REPL. It's "you inside your running program". You can poke around your code, draft it, debug it, right there. You can fire it on prod to understand a bug. Or use LiveBook, think of it like the Elixir version of Jupyter notebooks that can connect to your application.

- Real life performance is great, not because of speed but because of concurrency.

- The whole developer experience is great. Mix (the build tool, dependency manager, etc.) is simple, awesome. Dependencies are really rarely an issue.

- It's rock solid. In 3 years, I never had one downtime.

- LiveView is a god send. Not having to switch language for UI work is amazing, performance is great, and it's server side HTML which is amazing for SEO. My website is 99 on lighthouse without any crazy work.

- You need heavy computation and performance on some parts? Use rust, via rustler.

- You need to scale to multiple servers? It is distributed. Already. Just make sure to not have anti-patterns in your code.

- But the real kicker it's in its power due to the OTP platform. I think it's quite complex to grasp how much it's powerful when you haven't experienced it. Need to batch insert statements or rate limit api calls to a 3rd party service who can only accept one call per second per channel? A working simple solution is only 20 lines of code. Need to launch many different workflows running concurrently, keeping their state, recovering when crashing? 100 lines.

The exciting developments:

- Elixir NX ecosystem (NX, Bumblebee, etc): running and training AI models directly in Elixir, in a distributed way.

- Liveview Native and Elixir Desktop: two big initiative to bring Elixir to Desktop and Mobile applications.

- Gradual Typesystem. Jose Valim, the creator of Elixir, is working on that right now. I really liked the approach of set-theoretic types and the pragmatism of the approach. Hopefully it will be released in the not too distant future.

The "to improve":

I have the feeling that the platform (OTP) being the killer app per se of Elixir, the whole marketing of the ecosystem if 100% targeted towards developers. Which is good in many ways. But for the ecosystem to grow I think more initiatives towards business-type applications would be welcome. By example, there is only a few payment gateways libraries existing which is for me a sign of the lack of business audience.

Conclusion: Elixir made me a better developer, but most importantly a really productive one.

Some dumb questions: for liveview web / liveview native, is it possible to save rendered code offline and every time it connects to server , it will first check a defined version number ,if the version changes , it will update the code. If the the version is the same or if there is no internet connection, it will use stored code. Perhaps Elixi can be used this way for offline apps while maintaining the dynamic server rendering feature

In the context of LiveView Native / Desktop the renderer is in the client. So this part is ok.

The tricky part is that Elixir is compiled, and liveview templates are not "files" but functions.

From here you have two choices:

- You create a set of primitives (blocks), and the layout/arrangement of these primitives is configurable via a config map.

- Or you do a hot-code reload: Call an api, check if new code is available, fetch it, compile it (Code.compile_string), load it (:code.load_binary ), swap it for long running processes ( GenServer.code_change callback by example ), and remove old code ( :code.purge )

The first solution is safer. The second solution is a kind of black magic and you would need a way to authenticate de source code before compiling.

I don't think that would work - with Liveview the code is on the server and only the rendered HTML (or the dynamic changes) are sent to the client. Based on what you are suggesting, you'd need JS to to the processing or maybe WASM.

This is very inspiring. I'm in a similar situation! I'm also the only tech person for a small ecomm company, I'm just not the owner :)

I obviously have lots of questions but biggest one that relates to other convos I had on HN recently: did you use an off-the-shelf ecomm solution or roll your own?

It started with an off the shelf solution initially, but it didn't scale. I replaced 90% of it with our Elixir solution. The remaining 10% is the one that takes the longer to kill!

What was the off-the-shelf one if you don't mind my asking? It's total curiosity as I'm wondering what people go for in the Elixir space. Although it sounds like you started without Elixir?

It was initially just a little side business of my wife and I. We didn't expect it to grow as it did.

So I started with a PHP solution ( Craft CMS + Commerce ) that was flexible so I can just focus on front end part.

But then Covid happened and our business boomed super quickly ( in a month we went: from 2 to 10 people and 2 to 450 deliveries ). I needed to build features quickly to handle our fulfillment processes and built a separate stack of tools in Elixir, initially for packaging lists, routing calculations, sticker printing , etc... I still have some very fond memories of these crazy times.

Then I needed to redo our front end. We went from like 50 products to 1000+ products. I needed great search, smarter default sorting, recommendations, etc... and I needed the whole stack to be fast. I rebuilt everything with Phoenix and Liveview, building a weird layer to map Ecto to the super complicated Craft db schema.

We are using Craft just for the cart and payment now. The only pages served by Craft are the payment details and thank you page.

But I'm working slowly on building a complete E-Commerce stack ( a very opinionated one ) that I'd like to open source once clean and proven.

Thanks so much for the reply!

Looking forward to hearing about a new ecomm lib :)

I'm also rolling my own as it's built for a specific use case. Not sure I will ever open source, that isn't on my mind at this point.

That's because HN actively promotes Elixir. Even simple posts with 3 points hit front page.

That sounds somewhat similar to the “I rewrote X - in Rust”, or “Why isn’t Linux rewritten in Rust?!” posts that get upvoted quite a lot, or used to.

Reply on news.ycombinator.com