An Interview with Jose Valim, Creator of Elixir
welcometothejungle.co
welcometothejungle.co
Elixir is a very cool and powerful language for building web apps.
Having built my first product (https://retrochat.xyz) in Elixir, I can wholeheartedly recommend it to anyone who is not happy with their current backend language(Ruby, Python, Js, etc).
Docs, functional programming, OTP, ecosystem(Phoenix, Ecto, Mix) are really top notch. In my opinion, you will be able to reach much higher productivity (especially long term) in Elixir than in Ruby, which is known for it's high level of productivity. Sounds like a fairy tale, right? Well, that fairy tale can become a reality with Elixir.
The only thing that is a bit of pain is a good release support, but that has been changing for better.
We use a multistage Dockerfile to build the release and run it on a barebone alpine image, and it's never given me a single issue since I set this system up almost 2 years ago when Distillery was still the release library of choice.
And it's never crashed once, but that's the magic of OTP :-)
1. The stuff that never goes down (gets automatically restarted with last-known good state) and the amazing runtime in general. Not to mention it's much faster than Ruby or Python or PHP... In some cases it's faster than Javascript although JS and Erlang/Elixir are, on average, tied on performance.
2. A switch in philosophy and the way you troubleshoot your app: from magic/implicit to explicit. Many people cannot cope with this one because the force of habit is extremely hard to overcome. I for one enjoy knowing what exactly is going on in my app and exactly where but many others disagree. Again, it's a philosophical thing; you either are into it or you are not. There is also the objectively technological aspect: zero magic and implicitness mean much easier bug fighting. You no longer have to dig around in callbacks and installed on-start handlers.
Make no mistake: it's harder to start with an FP language. Nobody is disputing that. The benefits to be sown after the initial culture and brain shocks however are a lot.
I'll dispute it. For me, the functional paradigm clicked much easier than OOP and it's zillion abstrations and patterns. I'd argue that the best thing about functional programming is it's actually easier to reason about.
After that happened however, FP became much easier and much more pleasant to work with.
By what metric? Phoenix on Elixir doesn't perform any better than Ruby and a similarly lightweight framework like Roda: https://www.techempower.com/benchmarks/#section=data-r18&hw=...
NodeJS is also much faster than Elixir in other circumstances thanks to the V8 JIT.
Elixir's and Erlang's strengths over other dynamic languages aren't really performance. It's mostly just that Phoenix performs much better than Rails.
DB pool size is very subjective much dependent on how long a connection is held without making a query. A recent PR for Phoenix significantly improved performance by _decreasing_ the pool size: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153
I meant more like, people were using pools of 10 on very beefy servers and were surprised at bad performance. I agree with the PR author that the DB pool should be anywhere from one to double the CPU cores.
Thanks for the pointer. Currently reading the docs. Wonder if I can contribute to another language (not just Elixir).
I am ofc not saying it is a good thing from the elixir community. Just that it is the sad state of the momentum to change it right now.
For instance, I implemented a HTTP "business proxy" last year, which needs to be able to keep a large number of concurrent connections & proxy the call to a slow responding remote server.
I made multiple implementations (including Go, Node, Ruby with Roda, JRuby, Elixir, Crystal).
In this specific case I had, Elixir throughput & stability was by far the best. The gap (under load) between min & max was also the best (all this without tuning), and so was the CPU use.
Of course, YMMV, make your own experiences!
This type of topic is too prone to endless debates on the validity of this and that, and the fact that anyone proficient in whichever stack I tested could definitely improve it (including me on the subpart of those stacks which I'm most proficient with!).
I don't want to lose any energy with this, and I have made my decision for that business case (in this case, a very happy one).
You made an extraordinary claim, yet refuse to provide any evidence. Either backup your claims with data or don't make blanket statement perhaps?
As to your points regarding scrutiny.. that's the whole point. How else would be able to measure the metrics?
While providing little details, it is actually providing value to some here.
In terms of measuring success & picking stacks, I have also a good experience (after 35 years of coding - I hate to mention that and this is no "call to authority", but I've seen a fair bit of stacks), and Elixir is a good investment at the moment for my use-cases.
That being said, you can always release the repos and just archive them right away -- thus disallowing PRs, issues and any public commentary in general. And whoever wants to dabble further can just fork and refine them.
It's your call of course. But this is what I would do I weren't interested in further changing the repositories.
You are correct in your specification though; I should have said Phoenix in particular and web development in general, not Elixir itself. The BEAM languages aren't excellent in raw performance, that much is true.
Also I am yet to be impressed by JS web apps. They are quick when you spin them off on your local machine or on a world-unknown website that only you and 3 buddies visit. But I've seen NodeJS apps crumble at 10 reqs/sec routinely. I'll grant you that this was 2 years ago though.
---
In short, I am not claiming to have made a scientific and thorough research in ideal conditions. I did work with PHP and Ruby for a long time though (and have been partially exposed to Python), and the difference in performance and reliability between them and Elixir is night and day. I've seen an Elixir app with a rudimentary 20-tables data schema endure 1000 reqs/sec on a RPi 3B.
---
Final note: I'd like to see TechEmpower's benchmarks based off of a more real-world app like https://github.com/gothinkster/realworld as well though.
The Elixir/Phoenix app rewrite was a little more than 10x faster, much more consistent in response times, and needed far fewer resources to run.
Elixir and BEAM are clearly enormously better at some tasks but general performance is actually very similar between Elixir, Ruby, and Python if you use similarly performant frameworks.
I think, when it comes to web apps specifically, there are a few advantages that make a practical difference. These seem to be the major ones:
- Compiled templates and more efficient string interpolation
- Improved parallelism and lighter weight processes. This leads to better handling of many requests in parallel (esp. while most are just waiting on IO), better db connection pooling, less blocking, etc.
A few other ideas are mentioned here by someone who seems to know more about this than I do: https://news.ycombinator.com/item?id=12014243
You rewrote it.
The language you rewrote it in probably has less to do with it than rebuilding all at once what initially was built over time/organically.
But in this case the django app was a full rewrite of a php app. It’s quite similar to the eventual elixir app.
I feel it’s pretty much a fair fight, but you’ll have to take my word for it.
That's a very unfair blanket statement.
There are no ways to reduce the inherent flaws of mutable imperative languages no matter how much times you might rewrite the apps in the same language. You definitely can reduce bug surface but not by much; shared data is much more prone to bugs and human mistakes than a system with a runtime that puts immutable data and message-passing front and center.
I'll give you that rewrites do improve state of any random app. But I think you are overestimating this factor with your pretty broad and somewhat dismissive statement.
The measured workflows in terms of web frameworks include a lot of other elements and none of them are CPU-operations-per-watt (namely pure muscle).
Erlang's (and thus Elixir's) runtime shines particularly well in web apps' typical workflow, not on tasks where C++ likely is still king.
I’ve just shown you that it’s not faster. Benchmark Roda against Phoenix and you’ll see exactly the same is also true for the web workloads.
There is no magic.
(I'll again reiterate that I rewrote a Rails app from scratch to Phoenix and thus accelerated it at least by a factor of 20x. I am just a working guy and have no vested interest in spreading misinformation.)
A lot of web frameworks crumble or introduce abhorrent lags after a certain threshold of pressure. Elixir/Phoenix in comparison almost don't change their average response time until much later.
Truthfully, after reviewing the TechEmpower repo for the last day and a half, I can only see their Fortunes benchmarks as being half-adequate. Most of their benchmarks are extremely unrealistic in terms of web apps usage. At this point I am not even sure what their goal is anymore.
Elixir (Erlang rather) does have some "secret sauce" that makes it performant on part of the web framework space. For example, the literal pool used by the compiler+VM alongside iolists+`writev`-based operations make it so most template renderings in Phoenix allocate as little memory as possible. This means that any literal string in your template is allocated once on boot and not per-rendering, and building the output buffer of the template engine does not concatenate strings either (which would generate even more garbage).
These two blog posts have a bit more detail:
* https://www.bignerdranch.com/blog/elixir-and-io-lists-part-1-building-output-efficiently/
* https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/
So in a sense, everything is doable in any of them, but in Erlang this is the default way to do it, so it pushes towards an efficient approach to work with I/O from day 1. Case in point: I didn't invent any of this, I just learned what was there. But you won't see those differences unless you are working on an application that is rendering medium to large-sized templates (which are most apps returning HTML) and benchmarks don't tend to exercise that.
Other benefits from Erlang that may shine in the web space is the per-process garbage collection. It may reduce the variance on the latency and for short-lived requests, you may not perform garbage collection at all. All the VM does is to reclaim the space once the request is over.
At the same time, there are benefits in Ruby and Python you won't find in Elixir. For example, if you need an algorithm that relies heavily in mutability, then Ruby and Python will definitely have an edge. Discord recently had an example of where they made an algorithm much faster by moving part of it to Rust to leverage mutability. But benchmarks don't tend to exercise that either.
In fact, benchmarks may highlight non-ideal behaviour. For fully IO-based exercises, I got faster results by running an acceptor per thread that reads+writes as fast as possible, rather than multiplexing requests on all cores. But in practice, most technologies that can leverage multi-core will prefer to multiplex because that will be beneficial as soon as you do any subsequent I/O or CPU work.
TL;DR - sure Elixir can be 10x faster in some workloads, sure Python can be 10x faster in others. But of course, it would be incorrect to use any of these results to say Elixir or Python is 10x faster than the other.
*Fun fact: I am working on some VM trickery that makes things like Fib/Fac 3-4x faster but at the moment the trickery fails to show any benefit on any slightly more complex code. If they were to accept it, your factorial benchmark would show something much faster for Elixir, but in practice very misleading. :)
This one was a big draw for me. It was a major reason I got interested in Elixir in the first place instead of going the Scala/Akka route.
It's already the standard way in Ruby, even in Rails. Jeremy Evans did the template optimization you're describing using frozen strings 3+ years ago in Erubi and it was adopted by Rails just a few months later.
There's no long STW GC phase during requests in modern CRuby either. It's nowhere near as elegant as BEAM's heap-per-process design but the GC issues seen in CRuby < 2.2 have been gone for almost 5 years now.
GitHub retired OOBGC last year. Pause times are just 1-2ms and total time spent in GC is 0.5% on for the main application I work on.
Are you talking about generating better bytecode based on the semantics of Elixir or a bytecode optimization pass?
Yes, I was aware of the frozen strings optimization. Thanks for confirming! The part I am not fully aware though is in regards to the output buffer, which is a separate optimization. Is the output of a template being stored in an array that is writev-ed to the socket? Or is a template still rendered to a large string? Or maybe something in the middle?
If you don't mind me asking, what is OOBC? My Google-fu failed me. In any case, 0.5% is quite good, although that is also dependent on how much garbage the app generates. And as you know some libs/frameworks tend to be more mindful of that than others. :)
> Are you talking about generating better bytecode based on the semantics of Elixir or a bytecode optimization pass?
Do you mean the VM optimizations I was working on? I was mostly AOT compiling the bytecode of a function. That removes the VM dispatch overhead which further helps with CPU speculation. I see very good results on recursive functions but not much beyond that.
Ahhh sorry I meant Out Of Band GC.
> Do you mean the VM optimizations I was working on? I was mostly AOT compiling the bytecode of a function. That removes the VM dispatch overhead which further helps with CPU speculation. I see very good results on recursive functions but not much beyond that.
It's extremely difficult to get a good return on this approach for two main reasons: instruction dispatch overhead is a very small part of the VM overhead for VMs like BEAM or YARV https://www.sba-research.org/wp-content/uploads/publications... and Haswell onwards have no problems branch predicting the dispatch loop https://hal.inria.fr/hal-01100647/document
Sadly this is also why the CRuby 2.6 JIT isn't very effective either.
V8 removed their baseline JIT because it wasn't worth it. Firefox is considering doing the same. JSC has a more complex but effective baseline JIT. Implementing one for BEAM would be quite a lot of work compared to a minor performance improvement.
foo
<%= expr_that_returns_hello() %>
bar
output_buffer = ""
output_buffer << "foo"
output_buffer << expr_that_returns_hello()
output_buffer << "bar"
Now imagine the output_buffer is an array, it means less copying, but support needs to be added upstream and other places to handle them. You would also need writev support for arrays (or at least convert them to an iovec or similar).
Does this make sense or am I missing something obvious?
Btw, sorry for all of the questions and thanks for the linked papers. Now I have something to read on the weekend. :)
CRuby makes a bunch of memcpy calls but compared to everything else, they're fast enough they don't seem to matter much.
I've run perf on CRuby running a bunch of different web app type tasks (even rendering huge JSON payloads) and it's never shown up. You can see an example here: https://github.com/k0kubun/railsbench#perf
Perf output for when running Roda looks similar but vm_exec_core and gc_sweep_step are even more of a bottleneck.
This is because sharing reduces the memory usage in more than half. Without sharing (but with frozen strings), the memory allocation is `1 * static_size + 2 * dynamic_size`. I.e. the cost is each static part, which is copied, and twice the dynamic size, which is created and then copied (I am assuming that most of the dynamic content is allocated on the request life-cycle).
With sharing, the cost becomes `0 * static_size + 1 * dynamic_size`. Eliminating the static allocation is particularly important in loops. If you have something like:
<% for ... %>
pre
<%= dynamic %>
post
<% end %>
You could say that if you use an array or rope, there is still an allocation as the rope or array grows, but that is generally much smaller than the strings being concatenated. It can be preallocated too if you compute the amount of static and dynamic parts when the template is compiled.
So while memcpy is fast enough that it wouldn't show during profiling (except maybe by increased GC times), you should see measurable benefits in terms of memory usage.
Unfortunately I cannot prove this in practice in the context of Ruby, so I will gladly accept if you think I am overplaying the value of the optimization. :)
EDIT: taking a glance at the TruffleRuby rope paper, it seems templating with erb is around 3x faster once they moved to ropes.
Simple case: Adding a new column to a table Phoenix: 1. Migration 2. update schema 3. update changeset Rails: 1. Migration
Complex case: Add a new column with validation only on update Phoenix: 1. Migration 2. update schema 3. update changeset Rails: 1. Migration 2. add validation with only on update 3. pray it won't break anything in other parts of app :/
The complex case sounds like a simple one, but in a big codebase adding validations could be troublesome for a fee reasons, but let's not dig into them. In Phoenix you can have multiple changeset definitions with different validations for each of them, which makes it much easier to control when and what data is validated.
It's just one of the examples, there are many others...
I'd say, if you are content with Rails, it's great then! No reason to change a technology stack, because some random person is saying it's worth it. If you found a good enough solution for your problems, stick to it :)
But then again, who does not write unit tests to get 100% coverage against 110% of possible user inputs, in case you made a typo somewhere. Right ? Right ? </sarcasm>
Interestingly, coming to Elixir from another dynamically typed language (Python), I tend to do much fewer runtime/typing errors than ever before. I might be more experienced, or the tooling and compiler are good enough to catch the most common bugs.
I get a few "no matching clause" with my data ingestion tool on my Sentry log because external APIs are flaky and a cosmic ray flipped a bit so that I'm reading incorrect data. A GenServer somewhere crashes, restarts and the data comes back clean, without any user disruption at all.
But again, I'm completely ready to hear that it's just my setup, or how we failed to build the app.
Just wanted to confirm that it's completely possible to shoot yourself in the foot, which I'm surprised hardly anyone mention when talking about elixir.
Anyone interested should give it a try.
http://erlang.org/doc/design_principles/sup_princ.html#maxim...
"Let it crash" is commonly misunderstood. It doesn't mean that you shouldn't treat the known unknown cases of your business (like bad input), it means errors that are generally not known a prior (a database crash or network partition, for example) does not need to have a treatment since the Elixir/Erlang will simple crash and restart.
Things like K8s also brings a similar thing that Elixir brings to you, however having this in language is powerful since it allows ulyou to treat it better, if you want.
Your comments read like "but I was promised magic and I found no magic" to me.
Well, of course there's no magic. What there is is a really good runtime and a good tooling that allows you to avoid some mistakes preliminarily.
Maybe try VS Code with the Elixir LS plugin. It runs dialyzer automatically and highlights errors which also catches a ton of those runtime type issues. It’s pretty nice, often more expressive than standard typed languages.
However, this is not my experience. When something bad happens in my situations, it's usually in response to a bad input that is going to be repeated a couple milliseconds later. So the app just crashes in loops.
I don't understand how the "let it crash" philosophy is supposed to happen when the only thing that can happen after a crash is... Another crash. Maybe my case is just too strange.
There's no code in the world that can automagically recover from unexpected and not-coded-for user input, though.
I think it boils down to this:
> Here I'm classifying bohrbugs as repeatable, and heisenbugs as transient.
> If you have bohrbugs in your system's core features, they should usually be very easy to find before reaching production.
I don't know about "bohrbugs", but the vast majority of my bugs are "i-am-an-idiot-bugs", and boil down to "some combination of the handling one of the many possible inputs is going to trigger some code path where a function does not have the proper arity because I made a typo somewhere because I'm an idiot."
I claim that those are not "easy to find before reaching production" as a human being. (And I argue a large part of them are trivially found by a decent type static analyzer. Which, unfortunately, dialyzer was not last time I checked, 2 years ago.)
> By virtue of being repeatable, and often on a critical path, you should encounter them sooner or later, and fix them before shipping.
I disagree with that. My bugs happen on critical paths, but with non obvious shape of data. (I receive input from untrustworthy, aka "real-world", sources.)
I've spent enough time fighting bugs that were in the critical part of the code, were tested multiple times at the unit, integration and acceptance level, with different kind of simulators, but still broke on the first real-life usage because one function down a stack was not handling a nil or expected a string instead of a number, somewhere.
Encountering those issues "sooner" becomes a matter of "having enough imagination when writing automated tests".
I've heard property testing could be a tool to "automate" such imagination. But I'd rather have something telling me I just made a typo ;)
> So really, how efficient is restarting as a strategy?
> Well for repeatable bugs on core features, restarting is useless.
Amen.
> if the feature is a thing very important to a very small amount of users, restarting won't do much. If it's a side-feature used by everyone, but to a degree they don't care much about, then restarting or ignoring the failure altogether can work well.
It's hard to not read this as "Let it crash" being a good philosophy - provided it does not really matter, if your application works ;) (I know this is not the intended message, but still...)
I had great expectations about this section of the article:
> > I like statically typed language, and I restart my daemon after unhandled exception. What is Erlang gonna win me on fault tolerance
> This question was asked to me on a forum where I was discussing programming stuff and discussing the Erlang model. I copied it verbatim because it's a great example of a question a lot of people ask when they hear about restarting and Erlang's features.
The author follows with question (which is, basically, my opinion), with a very insightful explanation of supervision tree and OTP, but unfortunately, does not seem to talk about the "statically typed" part.
Which is a shame ! Message-passing, supervision and OTP are the great aspects of Erlang.
They're extremely powerful when used appropriately (which, in my experience, mean "sparingly, and as a plumbing-layer over purely functional code".)
My whole frustration with Elixir is that I want static typing "behind" the boundaries of GenServers.
Sure, I understand "GenServer.send" can not be typed, in general, and I'm ready to do some level of input validation at the boundaries, but beyond that point, please, If I tell you to add two apples, and I mispelled the name of one of them, don't even let me run, okay ?
I also know that there are efforts to do that (alpaca & co), but they're just too early-stages for my taste.
As said earlier, if you're solving the kind of problems where you never those kind of "i-am-an-idiot-bugs", or you're just not as idiots as I am, you may not find this problematic. I do.
But then, for dramatic purpose, the article ends with this gem:
> At some point, a team member got a very expensive credit card bill for the logging service we were using to aggregate exceptions. That's when we took a look at it and saw the horror on the leftmost side of the diagram: we were generating between 500,000 to 1,200,000 exceptions a day! Holy cow, that was a lot. But was it? If the issue was a heisenbug, and our system was seeing, say 100,000 requests a second, what were the odds of it happening? Something between 1/17000 and 1/7000. Somewhat very frequent, but because it had no impact on service, we didn't notice it until the bandwidth and storage bill came through.
Really ? Where those 500,000 to 1,200,000 exceptions not blocking anyone ? Not wasting someone's time ? Not causing improper or delayed results ?
Or, worst, just slowly making people lose faith into your software, or in software in general ? (Okay, I'm being slighly grandiloquant here, time to wrap up.)
Thanks for the link.
There are certainly ways you can get run away reactions that eventually take down the otp tree regardless of your settings and I probably do more input validation than strictly called for my the erlang guide books but not spending all of my time worrying about those edge cases does have it's upside with out negatively impacting other clients.
Or is your cluster more of an agnostic broker / message pump that lets other systems do the logic ?
As far as helping prevent bugs, let it crash won't help. It is not a replacement for a type system either. A type system for Elixir would definitely help you prevent bugs (but unfortunately implementing one is challenging).
Let it crash is a mental model and a technique to handle bugs and avoid undefined behaviour when all other attempts to prevent them (be it via static types, testing, code reviews, etc) failed. Imagine some data validation functionality had a flaw and that causes bad data to reach a core component, would you rather:
1. have the core component continue functioning with said bad input, which may have unknown repercussions?
2. have the core component crash and restart itself with a known state?
So let it crash can be useful to guarantee your system is not going to venture into undefined behaviour.
I wrote a small example of this in a blog post about assertive code (http://blog.plataformatec.com.br/2014/09/writing-assertive-c...). If you compare the two get_token implementations, you are 100% correct that in the second implementation, if you receive bad input, the code will crash over and over again. But I would say that I prefer the crash, so I can think about how I want to handle said input, rather than have an implementation that works by accident (and the accident may be a happy or an unhappy one). Would I prefer to have foreseen the bad input in the first case? Of course! And do type systems help avoid undefined behaviour? Definitely! But not all of them. It would be great if we could have both.
On a larger scale, especially when building complex systems, you are going to interface with client/server, database, apis, etc and you will run into unexpected scenarios. You likely can't prevent, model, or foresee all of them, so "let it crash" works as a default behaviour to handle them. It is also a legit strategy for things you know that may happen, but the best way to handle them is to start over anyway. It provides a mental model for you to reason when things go wrong at each individual component, instead of trying to reason how an exception may affect the whole system.
I am not sure if this helps, so questions and feedback is appreciated!
Or, finding developers.
I actually completely agree that Elixir developers aren't many. What I observed however is that people who want programmers that know FP and can apply it to real world problems where teams of web devs can be reduced from 20 to 3... get treated like coding monkeys and not given much respect and creative freedom.
You can't replace a pile of clueless juniors banging their heads at the latest Node fashion with several experienced craftsmen and treat them exactly the same. It just doesn't work -- people get discouraged and leave. I've had that happen to three acquaintances of mine (and myself, although I don't dare including myself because, you know, our ego gets in the way of properly gauging our own competence).
I've been rejected with the generic "poor cultural fit" excuse enough times on interviews to realise that most managers and shareholders simply want soldiers blindly obeying orders and not captains that can successfully make a team pull through a tough challenge on their own terms as long as schedules and business priorities are respected.
So, to me, the problem is also in the attitude of the shot-callers -- not only in the scarcity or quality of the Elixir devs.
The pain in Elixir releases I've seen has nearly always been self-inflicted. You can just run "mix phx.server" on the server and have basically an identical deployment experience as if you were running Rails, Django or similar.
Releases are an option that brings some power you don't get in other languages (particularly if you're using Distillery, which makes hot upgrades possible).
I wrote a guide on Elixir deployment that covers a way to use the new mix releases and deploy to Render in minutes. It also links to guides covering a number of other patterns, including the one I currently use for Alchemist Camp. It might be useful for others feeling some of the pain you did in the past!
https://alchemist.camp/articles/elixir-releases-deployment-r...
If allowed me to learn the paradigm of functional programming and build systems quicker than ever before. Combined with the Phoenix framework, I prototyped a tool within a few days that would end up changing my life. I turned it into a SaaS business (called FeedbackPanda) with my partner, and it has been turning my life around for the better.
Thank you Jose for being such a key person in all of that without knowing it. Your insights and determination to leverage new tech combined with a proven software paradigm. Spot on!
It's really hard to figure out how useful it is because people always build up hype when it's not always worth it. All I ever see is how wonderful and magical elixir is and how it's literally life changing. I want to believe but I've seen technologies with lots of hype that didn't live up to it. Plus elixir is different enough that I have a really hard time jumping into it to try it out.
----
I'll also add that recently I had to build a backend for a long term project I've been working on. I seriously looked at elixir/phoenix, but from my research it didn't seem like the pros outweighed the cons. Like there are some technologies that are so useful they're worth the lack of a robust ecosystem or the steep learning curve that may be necessary. And I am not sure Elixir meets that criteria.
There's already existing technologies like python/ruby for fast development, java/groovy for stability and groovy for modern features, and finally golang which is really good for concurrency and building web related microservices. Golang in particular is nice because it solves issues like python's slowness and it's slimmer than java in many ways. And these solutions are already pretty good. It's not like it's all that hard to write a backend. Like the productivity gains from js to typescript are huge because js can be bad enough that there's a lot of room for improvement. Whereas how much faster and MORE productive can someone really be in elixir?
And I am not even going to get into functional programming which is a whole other beast. It requires significantly different style of thinking, making a lot of my experience less valuable, I don't think it solves solutions for an oop system that's well designed, and often times the implementation can be bad if the right tools aren't there. From my limited experience, FP provides some really powerful features, but at the builtin cost of being more rigid when I don't need it to be.
What if you did not have to choose ? It gives you all of that. There is no "steep learning curve". I had a sysadmin that never did anything functional pushing code to prod in less than a week. This is not a single case. This is the widely reported result across companies switching to elixir.
Regarding rigidity: quite the contrary. In my experience elixir and its take on FP are faaaaar less rigid than an oop one.
I am not going to bad talk any of the languages you offered. I will simply say that i disagree with basically every single one of the characteristics you offer for them.
Regarding how much productive... Well the most extreme example is whatsapp doing it all with 8 backend engineers. All of whatsapp. Recently RocketMortgage had equivalent results. In the case of rocketmortgage, it was wihtout any prior experience in the ecosystem. This was for Erlang, but all the companies using elixir report equivalent results. PagerDuty and Bleacher Report are probably the most visible one.
So in the end... do what work for you ? I do think you are missing the point a bit with your argument, which is what i will fight. But if other languages work for you, great for you :)
I am confused by this response. You disagree with remarks like go being faster than python?
Wait, what? Whatsapp had 25 backend engineers in 2016, and they use almost exclusively Erlang, not Elixir. They’ve said many times that the Erlang platform has allowed them to do a lot with a small team, but let’s not turn their story into an urban legend...
Source: https://www.youtube.com/watch?v=87lW4Llsj7E (2:20)
* No mutation support means that for some CPU-heavy tasks, it's not as performant as languages that allow mutation. Some optimized algorithms that rely on mutation can't be expressed directly in Elixir or Erlang, and instead need to be linked in through a foreign function interface.
* It's not on the JVM so we don't get to piggyback on the thousands of dev-years of VM optimization and ecosystem around it.
* It isn't invented at, supported at, and marketed by a FAANG-type company. Money moves mountains and Ericsson doesn't spend as much on Erlang/BEAM as, say, Google spends on Go.
* Comparison across types is not a runtime exception, so you need to guard against crap like `nil > 1 == true` yourself.
* Erlang standard library, which is idiomatic to use from Elixir in the same way of using Java classes from Clojure, is a bit of a junk drawer of tools. There's good stuff, but you may have to dig for it.
* Elixir still hasn't nailed the balance between creating small, independent pieces of code ("applications" in Erlang-speak) and wrangling them so they can be used as dependencies from monorepos, multi-repos, etc. Umbrella projects were an attempt, I don't like them.
* There is an impedance mismatch between Erlang/Elixir's configuration system and 12 Factor® ENV-style configs. (Full disclosure, I wrote a library for this: https://github.com/appcues/config_smuggler)
* The built-in Elixir code formatter removes trailing commas from multiline lists. I mean, come on!
Take Haskell for example, which is also immutable but doesn't require you to spin up processes to do computations. It handles CPU intensive tasks significantly better, but is a worse choice for highly concurrent low latency soft real-time apps because of the scheduler and garbage collector mechanisms.
Edit: Btw thanks for bringing up that comparison across types issue. Never realized that. What's your solution to it? I'd write guards on functions that filter input based on types, then have a final fallback with no pattern matching that throws an error explicitly.
I wasn’t positive if the erlang VM restricted resource access in anyway.
I’m not doing anything CPU intensive, just bringing over a library I’m not smart enough to build in elixir :)
It is a decent language, but it's development sucks. Maybe I'm missing something. I use it for Jenkins pipelines and it does not even has an editor with code completion. I must run it through an interpreter to find simple syntax errors, like a missing parenthesis. Looks like I'm developing in the 80's.
It happens all too often that I have to manually cast stuff because the compiler cannot deduct the type of an expression. Seriously, it feels like writing Java pre-1.5 (before Generics).
At the same time, your code might just compile, and you'd get a runtime error because you forgot to rename an attribute that you referenced, and due to some dynamic wizardry Groovy tries to retrieve it anyway.
It is the very opposite of the Erlang universe, where the more time I spent the more impressed I got. So really no comparison between the two, forget Groovy if you have more than a toy project. Java on the other hand... will soon have it's fibers thanks to Project Loom...
The eclipse plugin has a pretty good approach in that it underlines things that cannot be deterministically type checked (that is: it can't prove they are wrong, but it can't prove they are right). It does a fair bit of type inference, so once you get used to that it's reasonably low amount of pain that comes from this type of problem.
It's sad that so many people form their perception of Groovy from what they encounter inside Jenkins.
I do full application development with it inside Eclipse and it is fantastic. None of the above applies. It's basically the same as developing in Java just 30% of the code size.
Pros :
- Elixir is fun. It's hard to describe why, but it is. Piping is additive, so are genservers and supervisors (which lead to anti-patterns but still).
- Phoenix is well thought out, documentation is good. Plugs are so simple.
- Websocket support is great, obviously (and LiveView is a gift).
- Ecosystem is not that big, but mostly quality and the automatic hex docs is a great feature.
- Concurrency is obviously good and, most of all, only there when you need it (looking at you, Node).
- Speed.
- Coming from PHP, having the ability to easily run tasks outside the current request is so freeing and eliminates the need of many third party components.
- Testing is good.
- Mix is great.
Cons:
- I don't care about static vs dynamic types but I do miss the type hinting of PHP 7. Typespecs are helpful but not the same (and Dializer reports are not fun).
- As a follow up it's hard to beat the IDE support of popular languages. (the Elixir plugin for Jetbrains' IDEs is good though).
- Not anybody will like functional programming I guess (it did kick my butt at first)
- Phoenix doesn't provide that much. If I compare to Symfony (and I guess RoR, Django etc) it's really noticeable. Authorization/Authentication, Cache, Mails etc. Also I'm not a fan of the templating (again, compared to Symfony's Twig).
- Also Phoenix speed of development seems slower (I guess the team is small and also currently a lot of work goes on LiveView)
- What I would give to have the cross-build capability of languages like Go, or the simple deployment story of PHP etc.
- Obviously a smaller ecosystem and less support online (StackOverflow is not useful, better luck on the Elixir forum)
- Speed (for some things).
- I do miss the return keyword.
- I do miss constants (modules attributes are not a replacement).
- None of your colleagues want to use it (some people just hate VMs).
I don't know too much about Elixir, but for implementing web services just use the tech you know the best - these days you should be able to scale almost indefinitely regardless of the stack you choose.
But if you have a problem that resembles a telephone exchange (eg. no UI, lots of concurrent IO intensive - but not CPU intensive - tasks that have soft-realtime requirements) give Erlang a chance. Even if you end up using some other technology it will really influence your professional development in a good way.
I'd love to contribute, but I don't know how to phrase anything other than a useless rant.
And I would love for the contribution to boil down to anything else than:
> life without a static type checker is hard when you're writing a server. That needs to stay, like, on.
Which is, arguably, not necessarily a jab at Elixir itself.
Sorry.
But if you're angry at a programming language, you probably need a holiday :-)
I could probably write a novel about the things I like about Elixir, but here's what I'd caution about for someone looking at adopting it over another alternative.
- Elixir is not statically typed and probably won't be. If static types are a tool you value, Elixir's success typing and runtime assertions might feel inadequate.
- Pattern matching requires discipline to not create heinous coupling. Pattern matching is wonderfully convenient for control flow and helps a ton with readability and avoiding deep nesting in complex situations. It's so convenient, though, that it invites reaching through abstractions into data structures in ways that make code brittle. The language doesn't really provide any guards against this, so it requires care and policing to keep things manageable.
- Because OTP is so powerful, it's tempting to avoid industry standard tooling and "just do it in Elixir." This is great for moving fast, but has the risk of pushing problems down to the line to the point that bigger system rework might be necessary. Background processing is a good example here. In Elixir/OTP it's easy to do everything in memory, but once you start needing failover guarantees, or need split out worker processes from a web server for independent resource utilization, you may end up getting rid of the initial implementation and doing the work to move over to a setup similar to what you might using for Ruby or Node anyway.
Learning to be productive in Elixir isn't really a huge investment, even for someone with a strict OO background. It's pretty light conceptually compared to other FP languages, can be written in an imperative-ish style if needed, and has clear, simple patterns for code organization. For every place you would consider mutating, you tend to find primitives that make working with immutable data structures quite convenient.
Absent from these comparisons are the elephants in the web world: the JVM and .net. The JVM in particular is superior in my experience to Elixir in all respects: statically typed languages, battle tested servers and libraries, abundant documentation and discussions, top of the line performance, excellent tooling allowing rapid refactoring and evolutivity, etc...
Pros:
- Less magic. It's a lot easier to figure out what's going on under the hood. - Phoenix is an improved Rails. Ecto is more flexible, the generators make more sense and everything is better organized. - FP leads to cleaner code. The difference to a neatly organized Rails (with POROs & service objects) is not big though. - Tooling/Mix is excellent. - Elixirforum is a great place full of helpful people.
Cons:
- Small ecosystem and lower productivity. I often had to spend 1-2 days to write a feature where with Rails I'd download a gem and it's done. - Elixir gets hyped up due to BEAM, OTP, stability, fault tolerance... - however, apart from some very niche cases most of its features can be replicated with background queues, Kubernetes, etc. - Speed & hosting costs. Elixir is a bit faster than Ruby and the footprint is smaller. But a powerful VPS + CDN is so cheap nowadays that it doesn't matter much at my scale. - It's an absolute niche language. There aren't many tutorials, SO answers or solutions for small problems - you often have to figure it out on your own.
Overall I'd say that Elixir is fun to work with and ist well thought-through. At the end of the day I only care about shipping products though. Writing my admin backend in clean FP code is nice, but I rather download a gem like Administrate and be done with it - so I have more time to focus on my core features.
From a business perspective I don't see many cases for Elixir: A small shop is more productive with Rails/JS/Python due to the vast ecosystems - and a bigger company will have trouble finding enough devs.
Don't get me wrong, I love learning new languages/tools, but the cost of learning them deeply is a price that's hard to pay for when you have to get things done at an ever-increasing pace.
It have a somewhat steep learning curve with the ORM Ecto. Debugging Ecto, changeset, etc.. is not fun. Dumping whatever Ecto returns can be a puzzle because by default other than the main tables are not preloaded and it loaded with meta data. Also how you deal with Ecto and forms in phoenix is not clear, lots of gotchas. The common thing is that you have a form template that update/edit and new/create share. The problem here is if you don't know what you're doing one of your template will expect a changeset and the other will expect a list. I forgot which one gotta look into my code but it was an annoying gotcha.
The lack of things baked into Phoenix makes it a hunt to find packages/libraries to add features to it.
Examples are Laravel having rate limiting router built in and signed links built-in. I don't believe the html form is as good as other frameworks; can be at time confusing. I think while Phoenix may not be aiming to be full features as Laravel, I believe security measure such as rate limiter and signed links should be built in.
Deploying is also a learning curve. You need to compile Elixir in production server and then the folder layouts is different than development. So hunting for the root web folder of your application is a thing I did and learning about the layout of production.
Having said all that, I enjoy coding in Elixir and using Phoenix. It is fun getting better at it. While there aren't many thing baked in like Laravel, it made me more educated in the things I've taken for granted. I re-did the authentication system 3 times. I tried jwt, did more research and realize when to use session and when to use them.
I enjoy coding in Elixir more than PHP. I see myself building my side web projects in Elixir in the foreseeable future. I do recognize that it's a smaller community with less contributors so I hope this come off as a constructive criticism of something I love and know well enough.
I'm struggling to start any in-depth project. I can do stuff - make a small app - here and there. But the thought of starting something is so daunting and intimidating. The irony is that I built a small but working language as part of the compilers course in college. I feel like I know so much more know but can't bring myself to develop something that reflects that.
But it is also worth saying my first attempt (at Elixir) sucked and it demotivated me to work on the project for 6 to 8 months. It was only later I learned/figured out some of the answers or possible solutions to the problems I was having and that motivated me to work on it again. Once I asked my partners to let me work part-time on the language, I prepared myself it would have to be a long term effort (at least 2 years). Then people started using it and that in itself kept me going (and it has been like that for almost 9 years now).
In all fairness, I also do struggle with motivation when starting hobby projects, even more larger ones, and especially if most of the challenges have been addressed. Also what keeps me motivated changed throughout the years, so I try to be mindful of that.
After tinkering around in IEX and doing the Getting Started Guide, I'd do Dave Thomas's course (https://codestool.coding-gnome.com/courses/elixir-for-progra...) and read Elixir in Action by Sasa Juric. Dave Thomas's course will get you through the basics of the language and the important concepts step by step. Elixir in Action will go further into the concurrency model that makes Elixir unique.
Finally if you want to build a web app I'd go through the Phoenix Guides: https://hexdocs.pm/phoenix/overview.html#content . Those guides will give you a good example of what a real Elixir/Phoenix Web Application looks like using a database with pooling and request concurrency out of the box. The Programming Phoenix book will take that further (https://pragprog.com/book/phoenix14/programming-phoenix-1-4).
If you want more practice writing functions and algorithms I'd recommend Exercism's Elixir track: https://exercism.io/tracks/elixir or Project Euler (https://projecteuler.net).
I stay with Elixir. Thanks!
The tooling gets out of the way, so a newbie is less likely to run into roadblocks. IEX is the best REPL I've worked with. Phoenix is much simpler to reason about than similar frameworks like Rails. And the docs are clean and consistent (+ dark mode). It was really easy to navigate through the docs, try things out in IEX, and get things working. It's just modules and functions. There's no classes, base-classes, abstract-classes, inheritance, instantiations and what-not. Just modules and functions. I doubt there's a better way to learn back-end development (DBs, Queues, Concurrency, PubSub, Distributed Systems, etc.) than Elixir.
A buddy of mine has been getting into Elixir recently while going to school for CS. The school's curriculum is mostly C#, Java, PHP, and what-not. He says Elixir demotivates him from some of the homework because everything in Elixir would be way easier.
Just in case somebody didn't know that
See https://github.com/rusterlium/rustler
Discord especially has several real world use cases where they've implemented native code in Rust.
As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.
You're getting down voted mostly because of your tone (IMO) but the appeal if Elixir is just a reality of experience.
When I found it, I was 15 years in my career and when I dove in, learned it and saw how it addressed so many short term and long term issues that appear in application development...the balance was the closest thing I'd seen to perfect.
But if I hadn't experienced the pain, I would never have appreciated it all the little details.
All I ever see is people exclaiming how awesome it is and then the responses turn out to be underwhelming for various reasons. Like hot reloading isn't that special or unique anymore.
But that's an interesting talk. I didn't have time to go through the whole thing but the core of it seems to be the pros of erlang like BEAM's high concurrency advantages and uniformity of tools running on top of otp.
It's an interesting argument.
https://blog.codeship.com/comparing-elixir-go/
Hot reloading is very unique though (but not necessary for most use cases). There's no other stack outside of the BEAM that has an equivalent. A lot of what is offered by the BEAM isn't even possible in other stacks because it requires a collective of language decisions working together.
Think how much the code would be simpler and easier to reason about without caching. Or reducing the numbers of servers you need from 150 to 2 and with better performance [1]
[1] https://www.techworld.com/apps-wearables/how-elixir-helped-b...
Sorry to be blunt but this is nonsense, languages and stacks are just tools.
When I was working on banking systems the idea of using dynamic typed language would have been insane. Now that I’m in a two man’s shop working on low traffic web app I’m happily using Rails for the CRUD work. And if tomorrow I were to work on a public high traffic real-time app like Discord I would do it in Elixir. There is no such thing as your stack.
Coming from a php / node.js background it seems very overwhelming
The default elixir-lang getting started is pretty good. https://elixir-lang.org/getting-started/introduction.html
Some people prefer this community one https://elixirschool.com/en/
Anyhow, your characterization there I don't think is quite right. For one Java does not typically use a background job set up, because it is a first-class multithreaded runtime so it seems more likely prepared to use a threadpool for some concurrent processing before it say tosses things out to a task queue. Similarly as most java web servers are on a thread pool model for concurrency as opposed to async IO, you'll also have true in request concurrency for things like making a few calls to the DB and then collecting them. This thread pool is not well abstracted typically, and the mutable object model leads you to writing classically tricky multithreaded code with data races and the like.
For something like node where you are not really likely to be using threads, you are pretty hard tied to async IO as your method of concurrency. So you have a really easy job if all you're doing is async calls to say a DB with no need for data processing. Here once you need real computation you will have essentially no option other than using a task queue.
Then when it comes to python you have essentially the worst of all worlds. it's not easy to do concurrent IO or computation without some form of background processing.
I believe that when it comes to the elixir runtime the idea is that they have removed async IO from the mix and instead put up a concurrency API that is as simple to manage as async IO in something like node. Under the covers is all delegating out simple blocking code to a thread/process pool, but the immutable data model and API makes it so that data is not shared across the pool, trying to keep the classic concurrent mistakes hard to make.
Nothing that can be done in elixir seem to be impossible or even astoundingly hard in some other technology, but the unified API for all forms of concurrency seem like a nice way to make it a runtime that can handle many different types of workloads in a simple and cohesive way. That simple way also seeming to not need you to bring in things like a redis task queue where you generally would need to use it or some other external solution.
Sure, you may want to move your Java/Erlang jobs to Redis/SQS/etc for different reasons, but it is not your starting point. And given you were explicitly talking about CRUD apps, being able to have everything you need with only your app + database, without a need for Redis, workers, or tasks queues, leads to operational simplicity.
For instance you linked the spring application events integration with redis, while that is definitely an option it's not one you'd immediately reach for. Instead you'd more likely set up and use application events and/or domain events (if using spring data) handled locally with thread pool. The benefit here is that of course way less set up and complexity, but you can also handle these events sync or async.
Erlang/Elixir processes don't share memory (Actor model) like Java threads do, giving it a distinct advantage (see Akka for Actor model in Java).
But yeah, for heavy stuff even in Elixir, you might want to outsource that to a Job server (which there are many libraries in Elixir for this).
You also don't have to worry about performance and concurrency. I would actually argue that you have to worry less about performance and concurrency because you can go further without having to resort to cache layers, tuning, etc.
But ultimately, I don't disagree with you. If you are building CRUD apps, then any technology is likely fine. Perhaps there is a chance Phoenix will start to take a bigger pie of the "CRUD apps space" with LiveView (https://github.com/phoenixframework/phoenix_live_view). If you could produce CRUD apps at the pace you do today, but making them interactive and realtime is orders of magnitude simpler, would that interest you?
1. Phoenix's channels make communication between a frontend and backend pretty effortless. Create a channel, publish a message, and anyone listening to the channel picks it up. Communicating updates from the server to the client is simple. Communicating between clients is equally simple.
2. Elixir has great support for spawning and managing background processes. My go-to is Rails, and if I want to manage background work I have to use something like Sidekiq and Redis. It works, but Elixir is much nicer. I've written systems that need to manage heavy amounts of background work and I dream of rewriting them in Elixir.
Just a few data points that might be helpful.
It's like, okay, so there's the functional programming, the similarities to Ruby, the BEAM VM, the concurrency.... but what does it do for me? They're not exactly selling points on their own because other languages can compete with those things.
You make an app using Ruby on Rails, keep developing it, and it keeps growing. You monitor your traffic and it gets bigger, now you add more servers to handle the traffic (it costs you $20/server, now you need 4). A couple more developments, memory usage starts to grow also.. Now, you need to increase memory of your server.. but now you also need Websockets, so you add both and need to add more resources to handle the scale, and the cycle continues.
When you use Elixir/Erlang and it’s BEAM VM, it’s highly efficient and scalable out of the box, so you won’t need to do “much” of the above. A simplistic explanation but shows the point.
OK, so in real life maybe the costs can actually be an order of magnitude higher than this.
But for $60/$80 a month? Is like no money.
That is less than one hour of time for even a cheap developer including taxes and other overhead, or okay call it two hours a month of time if you want to be generous. So if Elixir instead of Rails (or, of course worse, _switching_ to Elixir from current Rails) takes more than, say, 24 hours of developer time in a year... you didn't save money.