The Human Side of Elixir
akoutmos.com
akoutmos.com
https://news.ycombinator.com/item?id=2825689
I've always wonder how the Erlang ecosystem would be different today if a digital giant like a Twitter/FB/Google had used Erlang as part of their core business.
Note: yes, I realize WhatsApp owned by FB is run on Erlang.
Giants use and experiment with a variety of languages.
Yeah, well, somewhere, in some department, $bigco probably uses any random language you can think of.
Therefore, if you are considering the performance of the platform, improvements to the runtime and virtual machine, package ecosystem, etc, lumping them together makes total sense. On the other hand, Clojure and Java are quite different languages, with complete different paradigms, so there is probably a ticker layer specifically dedicated to Clojure.
Erlang and Elixir are much closer to each other than Clojure and Java or even Scala and Java. The data structures are the same, the language constructs are mostly the same (different syntax but the same semantics), etc. You can call each other with no performance cost whatsoever. All of the companies investing on Erlang will directly impact Elixir and, if you want to make the Elixir compiler or runtime faster, odds are that this efforts goes to Erlang - it is one of the reasons I have ~100 PRs merged upstream, mostly focused on the compiler and/or performance.
So, once again, if your goal is to assess the healthy of the ecosystem or how many companies are investing on the platform, then they must be considered together. The contributions done by Ericsson, Whatsapp, Klarna, Erlang Solutions, etc. benefit all languages running on the Erlang VM. In fact, if someone asks me how they can financially contribute to Elixir, I will 100% direct them to the Erlang Ecosystem Foundation (https://erlef.org/), which focuses on domain areas around the ecosystem, rather than languages.
Since you seem to use Ruby as a reference, I don't expect the dynamics to be much different. Most companies and contributors are likely working directly on the Ruby VM and the runtime, rather than the language itself. However, because Ruby is the only (main?) language running on it, we don't tend to separate them. But in practice the language is only a small surface into the complexity that is its underlying platform and working on said platforms requires a completely different set of skills and technologies.
EDIT: this applies to the ecosystem too. For example, every Phoenix application is running a web server written in... Erlang. :) All Ecto database adapters are using the tcp modules provided by Erlang. The package manager that powers all languages running on the Erlang VM is written in Elixir. And so on. You don't need to know Erlang in order to write Elixir but, if for some reason you need to pick up, it is straightforward because of all of the above.
But what runs your Ruby code is the Ruby VM (or even the JVM). What runs Java and Clojure code is the JVM. And what runs Elixir code is the Erlang VM.
The relationship between Elixir and Erlang is much closer to, say, Rails and Ruby (i.e. Rails is built on Ruby's abstractions and semantics) than Ruby and C. It is not a good example but it makes much more sense than the analogy above.
First of all, let's talk about the memory model. The Erlang VM manages memory per process (where a process is a lightweight thread of execution inside the VM that runs concurrently and you can literally have millions of them). So for example, when you have a web request, you spawn a separate process for this web request and once the process terminates, all the memory allocated to it is reclaimed by the VM. This means you can serve a web request without triggering a garbage collector even once. And if you need to garbage collect, it is local per process, so no system wide pauses.
However, what happens if you want to share a large blob of memory between processes? If you copied it between each process it would be expensive, so in this case the VM employs reference counting.
One curious point is that the Erlang VM did not give allocated memory back to the OS until Erlang/OTP 22 (roughly two years ago). This was not much of a problem in the Erlang VM because you typically run only one instance of the VM per node, regardless if you have 4 or 32 cores. Given the issues you mentioned in Python/Ruby/Node, could it partially be because you would rather start 4 or 32 instances per node, so if one of them accumulates too much memory and doesn't give back, it starves the other ones? Or perhaps you would have background workers fighting each other? If you have links to said issues, please share them!
Implementation wise, the Erlang VM uses its own memory allocation library to fight fragmentation. You can learn more about it on the erts_alloc API (http://erlang.org/doc/man/erts_alloc.html) and there is also a relatively recent article on instrumentation (https://blog.erlang.org/Memory-instrumentation-in-OTP-21/).
But let's not forget, AWS SimpleDB was written in Erlang, as well as the first iteration of Facebook Messenger.
I invested a decent amount of time in learning Elixir and Erlang and understanding BEAM. My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment.
Besides, Django, Rails, Laravel, etc are awful in those same benchmarks but adoption is fine.
The selling points of Erlang/Elixir are far more than simply non-real world benchmarks.
I was told, from someone familiar with the project at the time, that the first iteration was done with ejabberd, which is written in Erlang, with the main purpose of bootstrapping it. Erlang was not really in consideration in the long term at the time. Not sure if it changes your take personally but hopefully adding some context helps. I am always glad to be corrected if wrong. :)
I wrote about Techempower in the past. In a nutshell, the Erlang runtime does plenty of assumptions that you would absolutely use in production but which become overhead in benchmarks like these. Previous comment here: https://news.ycombinator.com/item?id=22894191
> My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment.
Your experience was guided by a decent amount of time and I actually appreciate you invested in learning it before deciding it wasn't worth it. However, I doubt every big company had the same takeaway for two reasons:
1. I doubt every big company went through the same experience (i.e. have they learned about the ecosystem?) to mirror the takeaway
2. The every quantifier is provably false, since there is Facebook/Whatsapp, PepsiCo, Klarna, Ericsson, Nokia, Mozilla, Cisco, and others using the ecosystem in production and/or investing on it (plus at least another company from FAANG using it at multiple projects)
https://blog.discord.com/scaling-elixir-f9b8e1e7c29b?gi=ecb5...
Edit: That said, squeezing every last drop of performance out of a server by writing everything in Rust is something that a FANMAG can well afford and may make more sense at the 100mil+ concurrent users scale, but it wouldn't be economical to start a project this way without certainty the capacity would be used.
Plus, there is far less of an "ecosystem improvement" to be made with Scala since it's riding on the Java Runtime (which already had massive adoption and development prior to Twitter selecting it).
Sadly most of the tooling funding came from the other big Scala user, who favoured an invisible-yield-point style (which I'm pretty sure inspired the model that Kotlin now uses with suspend functions).
One of the many tragedies of Scala (still the best general-purpose language on the whole, IMO) is that the community formed these somewhat siloed ecosystems that were each doing things their own way and had their own stack of libraries that didn't really play nice with each other. The language actually makes it very easy to write your own adapters (which may be why industrial users didn't create a lot of pressure to stop all the nonsense), but political fallings-out usually meant that it took years before there were "blessed"/official ways to use X with Y.
https://venturebeat.com/2008/05/01/twitter-to-jump-off-ruby-...
Shopify does probably have better engineering though. They have an amazing process of 2 code reviewers per PR, and one code reviewer is from another team. They write tests for everything and deploy continuously to production.
We have a new guy who started a bit less than a month ago.
He’s now already writing functional Elixir (no pun intended) and contributing to big code base.
One of the first things I told him is that “there is no magic in elixir”. What you see is what you get.
If you understand that everything is literally a function - life is easy and using the language becomes fun.
Except when it's a macro :-D
But yes, the only headaches I’ve had have been with hard-to-follow macros.
That said, the Phoenix Framework has some well-chosen macros that are a delight to work with. Furthermore, all these macros are defined in your source when you create a new Phoenix project—I feel that over all there is remarkably little magic in Phoenix, and what magic there is is easily inspectable.
But try to peek under the hood of Phoenix.Helpers someday... And also, don't ask too many questions about how Ecto drivers work... I tried writing an ecto driver once and gave up. Props to the people who made the Sqlite3 driver.
It's not magic, it's a regular feature of functional programming languages.
No bot.
What's the point of a bot?
I type that way because I have some kind of mental condition that makes me try to make things easy to read for everyone.
2. This quote is kinda funny to me:
> Many of the engineers whom I work with, actively seek out Elixir specific opportunities because they enjoy the language and the run-time that much (I’ll be diving into specifically why a bit later in the post ). This is also supported by a recent StackOverflow developer survey where 68.2% of people working with Elixir, wanted to continue working with Elixir. For some comparisons, Go received a 67.9% rating and Javascript received a 66.8% rating.
It seems like OP is trying to make the point that Elixir is so special that engineers will actively seek out Elixir jobs ... and then cites stats that show that's true only 1% more than Go and Javascript.
Maybe I'm reading it wrong and OP is just trying to show that Elixir is only as loved as Go/JS, not better, but in any case the stat cited does not show Elixir is any more special than either of those other super popular languages.
(I'm currently an Elixir dev, btw. I like it well enough but I personally don't think it's enough better than Go or JS to justify switching when the community is still so small.)
Personally, I find Stackoverflow's surveys pretty useless. Python has 73% "loved", which tells me...what? That 73% of Python developers like Python. But what else do they have experience with? Is this just objectively "I like Python", or "I like Python more than X"? What is X? For those who don't "love" their language (which really just translates to "they have not expressed interest in continuing to develop with it", based on what SO gives us), is it because of another language capturing their imagination more? What languages are those, have they actually used it in the past in production, etc.
As a C++ dev, definitely Rust. It got almost everything right that is horrible with C++, while still maintaining similar power.
Perhaps I miscommunicated what I took away from the StackOverflow survey. The point that I was trying to make was that in relation to other programming languages in the survey, Elixir ranked high with regards to how loved it is. The StackOverflow survey is just one data point in addition to the others that I bring up and like many surveys has it's own issues (like WebAssembly being a compilation target as opposed to a programming language the people program in).
Don't want to derail the conversation, but shoot us an email at admin [at] jinso.com. We are happy to tell you about us and get to know who you are!
Writing specs is tedious and apparently not even encouraged for private functions.
And what's liberating is that there can be multiple types (including patterns) for each argument and the return value, and the spec can capture all those details. This combination of specificity and flexibility is unthinkable in most languages.
But I guess that's what code review is for!
To that point, GenServers as an abstraction are super powerful and when applicable, are an amazing tool. For example, being able to control the initialization of a supervision trees using :ignore in the init callback can be handy to run DB migrations as part of the application startup. Or when used in combination with a registry it can be useful to hold on to user data in a GenServer and access it atomically across your cluster.
Elixir gives you Task for one-offs, which is a superior abstraction over proc_lib, and has seamless integration with OTP things like supervision trees, which you will have to roll by hand and probably mess up if you use proc_lib.
For just stateful data, ets is much better performance-wise than a GenServer (with the tradeoff being complexity in understanding matchspecs and the like), and Agent is much simpler for dumb state and leads to better code (though I basically never use Agent).
Honestly, I think the GenServer behaviour is awful (not GenServers, themselves, they are great), because it does not engender writing well-organized code. Dave Thomas' empex talk about it is really eye-opening and you will understand the internalized pain. (I don't like his specific recommendations, and I think the ship has sailed on Elixir reimagining how GenServers are organized).
So yeah. My philosophy is don't write GenServers, unless you are really sure you need them (like maybe collaborative state for games; to wrap a stateful communication protocol, like http tcp, or an application layer over udp; or to to model something IRL or remote, e.g. browser state as in liveview, or as I have in a previous gig, qemu VM state). And even for some of these you're probably better of with a gen_statem or the like.
> supervision trees, process linking vs monitoring
you still need to understand these to use Task correctly, or gen_statem, but those are really advanced topics. It's crazy, but you can get really really far without those (even if it's not, strictly speaking best practice), and it's not like once you learn about them it's hard to change your code to start using those correctly.
Sometimes things take a little longer to write as you have to be slightly more verbose, but it's so much quicker to understand when coming back to it.
The pipe operator |> that you mention is great example because it does something in OO that is typically done using a 'fluent interface'[0] often used in 'Builder' classes (i.e., the class methods return the object so that it can be chained together). The great thing about the pipe operator is that it's much more simple and flexible, it simply passes the result of the prior function as the first argument to the second. This means, unlike with a fluent API, you can pipe together functions from completely different modules:
:crypto.hash(:md5, some_binary)
|> :base64.encode_to_string()
Elixir also has protocols[1], which allow a function to be used across a variety of types (similar to obj-c/Swift). Which again is more flexible, as the functionality can extend a 'class' that you don't control.Elixir also has behaviors[2], which just guarantee that a module contains a number of functions, something akin to interfaces Java.
Dialyzer can take advantage of behaviors and protocols to enforce typing constraints.
[0] https://en.wikipedia.org/wiki/Fluent_interface [1] https://hexdocs.pm/elixir/1.12/Protocol.html [2] https://hexdocs.pm/elixir/1.12/Behaviour.html
I learned F# in a ~year, and then found a Clojure job. I _never_ want to go back to OO.
Yes, I do enjoy FP on the long run, not only to write it but more importantly to maintain it (much easier in my opinion, including refactoring & tech debt reduction).
I still use OO too (e.g. with Ruby), but my Ruby style has gotten more FP-oriented as a result (e.g. ETL components which looks a lot like Elixir modules).
x, err := computeAThing(1, 1)
y, err := and_then.Int(err, makeComputeAThingFn(x, 1))
z, err := and_then.Int(err, makeComputeAthingFn(x, y))
another, err := and_then.AnotherType(err, makeComputeAnotherTypeFn(z))
where makeComputeAThingFn(...) returns a function with the body `return computeAThing(...)`. `err` will end up with the first non-nil error and the following and_then.X's will be no-ops.Sponsored content
I feel the same way as the author and really want to find a job where I can work with it full time, but my current work situation is too good to give up.