Two years of Elixir at The Outline | Hacker News Reader

Two years of Elixir at The Outline

blog.usejournal.com

236 pointsdavydog187104 comments

I'm bullish on Elixir and I agree that ExUnit deserves to be singled out.

Whether you consider Elixir fast or slow probably depends on what you're coming from and what you're doing. Personally, I wish the performance story would significantly improve.

But, the reason I think Elixir should be your goto language comes down to process isolation and the ecosystem around it (the synergy of the runtime, libraries and language). If you're doing a pure web app, you might not fully leverage it. But, for anything that involves long-lived in-process data (like a workflow/pipeline, or persistent connections), the way to approach this in Elixir (with supervisors, processes, message passing and pattern matching) tends to result in highly cohesive and maintainable code. Message passing is a pretty effective antidote to coupling.

I just finished writing code that had to take a stream of unordered time-based data and dedupe it over a small window. I'm not overly pleased with the final result, but it's completely isolated from the much larger app it sits in, easy to refactor/rewrite, with no risk of someone taking a shortcut and just accessing some internal detail, and testable.

I feel like I wrote a microservice, but without any of the challenges of a separate app.

I can see the logic behind the statement 'message passing is a pretty effective antidote to coupling' but I personally subscribe to the idea that actors don't compose. In light of that I think use of actors and messages implies some level of coupling which may not be obvious. Actors can't be arbitrarily composed together without knowing which actors they will be interacting with, making their addresses available and understanding their protocols so they can communicate together. Functions have no such requirements so I don't totally understand how message passing implies less coupling than conventional function-based architectures

Message passing is less coupled than functions or "objects" that share bits of state.

I think the pain-point being designated by the OP is that the cure is worse than the disease:

    Problem = tight coupling
    Solution = x, where x => {indirection, implicit_coupling}

I'm not sure I agree the final judgement call, but the analysis is spot-on. Message-passing interfaces are generally difficult to discover and to debug, and require a lot of careful documentation, logging & instrumentation.

So you create a function around the message. It's not uncommon at all to have something like (pseudocode)

myActorModule.increment(ref, amount) -> ref ! {increment, amount}

myActorModule.syncIncrement(ref, amount) -> ref ! {increment, amount, self()} receive -> ok timeout 1000.

You can build out the message passing interface to an actor -as- functions, and even supply sync ones if you want. The actor implementation can be completely opaque to the caller; just the exported interface functions are defined, exported, and supported.

> Message-passing interfaces are generally difficult to discover and to debug, and require a lot of careful documentation, logging & instrumentation.

You just use the "most common structures". For example, Elixir has the Agent module, which encapsulates state, but lets whatever you're using interact with the state via lambdas (or passed functions).

Another common structure is the state machine. I was somewhat displeased with the complexity of gen_statem, so I wrote my own! It's about 50 lines of code, and can have plugins to do facilities like logging and debugging via a plugin-able interface.

Ultimately the real win with message-passing is not having to write spinlocks, semaphores, or mutexes.

You can use functions without knowing their inputs and outputs, and your functions don't need environments that somehow allow them to lookup the other functions they're going to call?

Messages and function calls aren't quite isomorphic, but they're pretty close. (They get really close if you include coroutines and coroutine-like things, to cover the non-1-to-1 relationship between messages and replies that messages can have.) I'm not sure the difference can sustain the implication you're trying to draw. Especially if you include RPC in the function world.

I think you're assuming side-effect free functions (and apparently ones whose interfaces are automatically known, since the interface to a function is just like the message handling in an actor; in fact, Smalltalk, the original OO language, didn't have 'methods', it had 'message passing'). But, you can still write libs and share them in Elixir, to benefit from any argument you'd care to make re: functions.

Actors are all about managing state. It's when state starts leaking around or being modified unexpectedly ('wait, why is that being called there?!') that a more typical language (Java, say) starts to get tightly coupled, and sharing that state is the norm, and expected. And I agree, if you have pure functions, that issue is largely controlled for too. Just, that's a really hard paradigm to stay inside of and still be productive.

I think that's why microservices are even a thing; it forces you to decouple state from different 'functional' parts of your system; by creating separate systems you have to be very intentional about what state you share, and how you share it (database or redis or an API call or similar depending on need).

Actors do that automatically for you; the state is contained within them; there's no chance(1) it will accidentally be changed by some other process/code, and the 'controlling' process gets to decide what messages affect changes, and how those changes occur. Or, in other words, all state changes relating to a context happen in one place, rather than state being passed around to be changed anywhere willy nilly across contexts.

1 Technically there are system level things that allow you to munge state inside of an actor. This is more frowned upon than using reflection for arbitrary reasons in Java, however, so the only reason you'll ever do it is because you're debugging what's going on in something, or you have decided you absolutely -hate- your toes and need to footgun them off.

You can certainly achieve the same low coupling with anything. But ....

Messages tend to be lightweight. Often a symbol/atom "command", and a few scalar values. Like command line arguments. If the process needs more data, it sends its own message to the owner of said data.

With conventional functions and OO, you're often dealing with complex objects (with references to references ...) which often leads to breaking the Law of Demeter.

Message passing also gets your asynchronicity. You're also decoupled from specific runtime implementation (the receiving can be on a different machine, the semantics are the same). You can achieve the same without message passing, but it's certainly more baked-into the runtime/tooling/language.

Joe Armstrong's popular quote on this:

"The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle"

> I feel like I wrote a microservice, but without any of the challenges of an separate app

You feel that way because you did :)

I keep wanting to move to Elixir but always go back to Erlang. Any advice?

Speaking as someone more into Elixir than Erlang, I'm wondering why you want to move to Elixir at all? If you're happier in Erlang, then why not stick with that? For me, the best thing about Elixir is the BEAM, and you've obviously already got that. If you don't want or need metaprogramming, or the various Elixir conveniences feel less than convenient to you, it seems like you may as stick with what you find comfortable.

The advantages of Elixir over Erlang, for me, mostly come down to the modules in its stdlib and package ecosystem.

If you don't enjoy writing Elixir, you could always just install Elixir, and then write Erlang that calls into Elixir-stdlib modules.

I believe you can even install Elixir packages into an Erlang rebar project, if you like. No need for Mix or anything.

Why do you go back?

> Personally, I wish the performance story would significantly improve.

Before asking for performance improvements, have you tried:

1. tuning the BEAM? It has a lot of knobs. For example, there's "initial heap size", which can effectively turn any subset of short-lived processes that would otherwise need to do GC into simple "allocate an arena, do work, deallocate the arena" workloads.

2. tuning your OS? WhatsApp had to tune their (FreeBSD) kernel quite a lot to get it delivering packets to the Erlang runtime efficiently.

3. disabling tracing support? Part of the reason ERTS focuses on being a bytecode interpreter, rather than an AOT compiler, even in production, is that this allows ops engineers to connect to production nodes and trace/debug/profile code that's not doing what they'd expect, watching what happens as real user data flows through it. Because of this, the interpreter is "instrumented" even when a "production" (embedded, etc.) profile is used to generate a release. If you aren't going to use this tracing support, though, you can tell the runtime to globally disable it—for big wins in interpreter speed. (I separate this from #1, because this is basically the last-resort BEAM knob to turn. You're throwing away a lot of cool stuff by doing this.)

3. using HiPE? Specifically, sticking `@compile {:erl_opts, [native: :o3]}` into a subgraph of your modules (like a parser, a lexer, and the code that calls them both) can yield major improvements.

4. really using HiPE? I.e. recompiling both OTP and Elixir, and your project, using `ERL_COMPILER_OPTIONS="[native,{hipe, [o3]}]"`. This makes some things slower, so it's not the default, but it can beat #3 for modules that make a lot of stdlib calls (e.g. anything that loops using calls to Enum/:lists.)

4. using NIFs? (I don't recommend going this route until you've found that all of the above fail to get you enough performance, because the above all let you continue to just write plain Erlang/Elixir, while this step requires worrying about FFI. But it can be a big win, where and when it makes sense, just like in any other interpreted language.)

If your Erlang code is still slow after all of those, then I'll be surprised and would like to hear your specific story.

Thanks. This is really helpful. I tried to think of some specific example. I think most of what I'd generally come up with is tied to the available datastructures. We recently tested some code to essentially do a group-by in code, with various aggregation functions, and it was considerably slower in Elixir vs D and C. Very simplified, something like:

   Enum.reduce(rows, %{}, fn [user_id, hits], agg ->
     Map.update(agg, user_id, hits, &(&1 + hits))
   end)

I didn't apply any of your suggestions.

If I had another performance-like related pain point, it's throwing multiple cores at some data. I think it's somewhat common to have a large piece of data that you want to concurrently access. For example, keeping a large suffix tree in memory for an autocomplete endpoint, and letting each request concurrently access it. This isn't really a specific performance issue, but it's a performance sensitive pattern that is very antithetical to everything Erlang. ETS has limited structures, requires copying of values, and has no atomic replace (so you can't build a new version in a temp table and just rename it).

Tryout using Rustler and passing Refs. It’s a handy combination and Rust avoids most memory management issues one would face with NIF’s in C. I’m working on some data calculation code and think I’ll go that approach.

Specifically for the suffix tree in memory, you would store the tree as a constant in a module and dynamically recompile at runtime. By doing so it puts the tree in a special heap that can be read by all processes and doesn't get copied around. There's a library called 'mochiglobal' that does that for you. Discord ported it to Elixir and called it 'fastglobal' (although their implementation can be sped up by about ~25%).

I think most people would do 4 before 3 and even 2.

Most erlangers would not use HiPE in production.

So, just to speak about the BEAM, which is the part of Elixir I am familiar with, I don't think any BEAM language will ever be a good goto language. The BEAM isn't good at general purpose computing. It's not particularly efficient at performing calculations or using memory. It's good at reliably performing as you increase memory pressure on your system (via message passing). You can scale a business to a decent size on a smaller number of fatter nodes (high RAM machines). Connections will keep pouring in and you'll get a consistent ms.

That said, you'll run into issues once you hit a certain level of scale. I've had to make patches to the BEAM just to keep us scaling, and I've never seen someone get away with stock dist. You pretty much have to rewrite pieces of OTP.

Now is Elixir ok for a CRUD app serving 10G of traffic? Of course... but this is also the age in which every language has a solution to the C10K(, C100K, and more in many cases) problem, and there are compelling reasons to write your CRUD app in Go and get better library support, static types up front (trust me, you want a static type system), and lower latency.

Personally, I'm bullish about Rust. With async/await landing, it stands a good chance of providing the same focus on safety and reliability while being infinity more flexible at what it can effectively. It's a better general purpose language. Also, the actor model (and CSP as well) aren't the end all be all of concurrency patterns. Rust only provides the building block (futures) so it's much easier to choose the concurrent pattern that fits best for your problem. You can sting a future using the actor model, so it's still an option, but you aren't limited to it.

Ok... yet 99% of development is trying to write bug-free and readable code and do it pragmatically.

Elixir gives you that plus a lot of goodies that are really hard to get in other languages. The creators of Erlang were really clever and in my opinion, so are the creators of Elixir. If I ever were "bullish" on a language, Elixir would be it.

Elixir is a very pragmatic language. It doesn't get in your way, but still forces decent habits and design on your application.

> bug-free and readable code and do it pragmatically

Erlang has historically been a bad choice for this, and there's been a lot of effort (dialyzer, records, maps etc) to make it better.

Dynamically typed languages are, in my experience, not worth the cost of maintenance and readability. In Elixir and Erlang it's possible to create function signatures that tell you almost nothing about the shape of the data you expect to pass in. This makes it really hard to collaborate at scale. Various attempts to enforce dialyzer use are usually met with 'we need to move faster' or 'we'll add them later'.

So if you want to write bug-free readable code, I think Rust is a much better choice. It's statically typed, and the addition of traits and lifetimes tell you so much about how a function or type will behave, and there's no need to sync specs or docs. The code doesn't lie.

Exactly, this is why the most reliable piece of mission critical system was written in Erlang.

I've hardly run into any bugs in Elixir. It feels almost like Haskell to me. Pure magic.

On the contrary I struggle to get even the most basic Rust program to compile, wasting hours before anything is done.

just a nitpick, elixir does not let you write bug free code, it lets you write buggy code and actually get away with it.

Until someone re-implements the process isolation and other battle-hardened distributed systems characteristics of BEAM/OTP in a rust stack, it certainly seems like embedding Rust as NIFs into erlang/elixir seems like a nice option to make up for the perf deficiencies you mention (though doesn't cover the memory concerns.)

I have not tried doing it, but I'd imagine the safety guarantees of Rust are a nice thing to have to reduce the risks involved around NIF callouts. I believe Discord recently posted that they are doing this, and I'm interested in hearing about their results.

edit: Google is my friend. https://github.com/hansihe/rustler

I think they're being reimplemented in Kubernetes. Kubernetes provides scale-ability, fault tolerance, and the pieces you need to build robust distributed systems. There's no need for this from the language itself. It's actually disadvantageous to mix Erlang and Kubernetes, because Erlang really wants a stable set of node names.

It's not a good fit for the way in which we're doing fault tolerance in the cloud. Erlang focuses on not crashing the BEAM, by letting processes crash while allowing graceful degradation of service. Supervisors are akin to Kubernetes deployments. Processes are pods. The pods themselves should be designed to crash gracefully.

BEAM on Kubernetes is certainly possible: https://github.com/bitwalker/libcluster just not by default

In case it's of interest, Sonny Scroggin gave a talk last March at Code BEAM SF (i.e. the rebranded Erlang Factory) on writing NIFs in Rust, if you're interested:

https://www.youtube.com/watch?v=F86JXaCPowo

(Sonny is a core team-member of the Phoenix framework).

actually no you do need a static type system (i dont trust you on that one, i trust the extensive studies instead) and Go would be the worst example of static type systems anyways.

not having a static type system in elixir is great. I have my deploy code which takes a certain type of map, but i can paralellize my integration tests by adding a test_id field to the map which is passed a UUID in my HTTP call, which then goes stores in the API harness an RPC'd link (erlang RPC is nice enough to message-pass correct proc id's) to a dynamically created mock of the component which cannot be parallelized. At any given time during integration testing 5-10 of these mocks are in flight on the backend.

You could probably do this with a static type system. I wouldn't want to. I'm not an "expert" elixir programmer, and I got this up and running in 2 hours.

Oh I remember you - you were in the other Erlang thread peddling Rust exactly the same way. You'd make general claims of authority about Elixir/Erlang and how there were all sort of vague issues with them, and then gave out blanket advice for how Rust would be a better option.

It was pretty annoying to read.

What did you have to patch in OTP? Usually it's pretty easy to get your changes merged upstream... And there's plenty of companies running large Erlang projects without any custom patches.

> The BEAM isn't good at general purpose computing. It's not particularly efficient at performing calculations or using memory.

BEAM languages are fine, easily faster than Python, Ruby, PHP, and competitive with v8 in many cases. Furthermore anyone doing serious data processing in a web service uses a work queue, which can be serviced (if you so choose) by workers written in any language you like.

Ruby has had a VM competitive with BEAM since Ruby 1.9. BEAM has a slight edge but Ruby also has JRuby with a real JIT and all the GC goodness of the JVM.

Modern Python, Ruby, PHP7 are all much faster than generally expected outside of microbenchmarks.

once Pony matures might be a good option if you care about performance

> I'm not overly pleased with the final result, but it's completely isolated from the much larger app it sits in, easy to refactor/rewrite, with no risk of someone taking a shortcut and just accessing some internal detail, and testable.

The 'let it crash' is really something I've not encountered in any other programming language/environment.

I've got a constantly growing Elixir project that I use for scratching various itches. Quite often I'll prototype some new feature in some of my free time, and quite often what I come up with is pretty haphazard.

In some cases the stuff I have running crashes for the silliest of reasons, but the rest of the app is unaffected, so it doesn't matter. A look at my log often makes it clear that I forgot something as stupid as checking for a timeout to a HTTP request. I fix the issue, push it to the server, recompile in the IEx console (or Application.start()/stop()), and keep an eye on what happens next.

When I have a bit more time, I might actually take a more careful look at the code to fix some obvious issues that somehow haven't caused problems, or I might add some typespecs when things seem to have solidified, where Dialyzer/Dialyxir, via its 'success typing' (which I guess is basically gradual typing like TypeScript?) will probably help me avoid stupid mistakes down the road.

What I like is how organic the whole process feels. The godawful, badly thought out parts of my project don't cause the whole thing to crash, and the more solid parts can be augmented by a type system. It's an approach that I was previously entirely unfamiliar with.

> 90ms is a 90th percentile response time on The Outline. Our post page route is even faster! We got this performance out of the box

That's one of the features you get when using BEAM VM -- your code will likely be scalable out of the box as it's already built out of isolated independent small processes. If you want to handle more requests just run on a bigger machine.

> The Community is wonderful!

I use Erlang mostly but one major thing I admire about Elixir is it's community. Jose and team did a great job there. So it really has both - the great technical stuff, built on BEAM VM which was battle tested for decades, and the great community and tooling.

> As soon as you get data from the external world, cast it into a well known shape.

Good advice. Have sane validation at the edges and then convert data to some internal format that's easy to handle. As opposed to defensively sprinkling validation conditionals everywhere in the core code. That's makes the code cleaner and easier to reason about.

As a long time Erlang developer I was suprised when I first heard of Elixir, and even more surprised at how nice it was.

That said I think Elixir is very unerlangish. It’s very much Ruby ported to a new platform, it’s heavy on meta programming with macros and using a giant solve all your problems web framework (Phoenix). Compared to Ruby on Rails I think it makes a lot of good arguments. It’s faster, mix is a better package manager, and the BEAM is reliable at small to medium scale.

Of course I recently posted about how I was moving away from the BEAM to Rust. I think Elixir has reached peak adoption, and I think the days of large frameworks like Rails and Phoenix that ask you to program in a DSL are sunsetting in favor of client side frameworks and smaller web services built off of thinks like Go’s net/http.

I also think the reliability of BEAM is happening at the level of orchestration with containers and serverless. The BEAM itself doesn’t fit well into the cattle model of reliability and it has scalability issues when you reach a certain cluster size.

Erlang is also quite slow at processing data and performing computations. What it does provide is reliable latency and load as you scale, which lets you grow your build mess quickly without buying more hardware or performing software miracles.

I’m more optimistic about Rust because it provides the reliability I’ve come to expect from Erlang. We’ve been able to scale Rust to handle the same load as our Erlang cluster with a more manageable code base, lower average client latency, and lower memory usage on the same number of machines.

I also know that people like to treat dynamic vs static typing as a matter of taste, but in my experience dynamic typing only works until you reach a certain level of scale. I’ve watched several python and erlang codebases turn into incompressible messes because of this. This is why we’re seeing things like typescript and mypy become requirements rather than nice to have. So I’d rather just bite the bullet and get a really nice static type system up front.

>>I think Elixir has reached peak adoption, and I think the days of large frameworks like Rails and Phoenix that ask you to program in a DSL are sunsetting in favor of client side frameworks and smaller web services built off of thinks like Go’s net/http.

Is this just a hunch or do you have numbers to back it up?

edit: This is the second Elixir/Erlang thread you have attempted to hijack by spreading FUD about them and pitching Rust as an alternative. You come across as a Rust shill.

You don't have to use Phoenix... I'm currently doing a plug-only deploy and it feels great.

> I’m more optimistic about Rust because it provides the reliability I’ve come to expect from Erlang.

Im very curious about rust myself, Out of curiosity, how does Rust protect you against wierd things happening like errors in drivers, or dropped packets, e.g.

Phoenix is really a very small framework, and its codebase can be easily understood in just a couple of days. Phoenix being "large" is a common (but unfounded) criticism that seems to stem from Elixir's superficial resemblance to Ruby.

I’ve read the book on it, and I disagree. It’s certainly its own platform and you’ll have to deal with its ups and downs and learn its DSL.

The only way Elixir resembles Ruby is the syntax. Everything else is totally different.

The similarities are exclusively because of the do..end

Beyond that there's barely any resemblance.

The optional parenthesis syntax is pretty similar and arguably a bit unfortunate, though wider adoption of the code formatter seems to largely correct this. It's nice in the console to type a bit less, but I can't imagine the utility of this in code that any other person is going to read.

Often times when writing a macro or a function that should appear as though it's a part of a DSL, the advantage of turning off parenthesis can shine here. That's why parenthesis are optional with a lot of the Ecto macros.

I agree. As an additional point, when you have auto-formatting on save, this can also enrich your insight about the DSL at hand. It has for me.

Optional parentheses are pretty much a requirement considering how much of Elixir is actually 'just' macros. I'm very happy that the formatter adds them for me as much as possible though, because I'm also not a fan of how far Ruby takes it.

Agreed. I used to remove parenthesis as often as possible. Looking back at the code, that may not have been the brightest idea.

I really do like the coding convention though that lack of parenthesis can indicate a function meant to be a part of a DSL.

As a long-time Emacs user recently I finally bit the bullet and started working with VSCode (with vscode-elixir and ElixirLS installed).

...Gods, what a relief. Especially the automatic formatting on save is just a lifesaver. At certain point I would not mind if there is one formal syntax enforced like Go does.

Ecto is practically the only library I use where I don't pay attention to parentheses. And the formatter has me covered afterwards.

I was amazed to see how VSCode will suggest typespecs all over the place!

I mean, I read the codebase, which is why I'm saying it's small and easily understood. Sure, it has some specific things to know, and some ("magical") helpers like automatic view functions. But it's no more its own platform than any other micro-framework.

I'd be interested to know what you believe the ups and downs are, aside from speed.

Criticizing Phoenix for being the Rails of the Elixir ecosystem is particularly ironic given Phoenix's creation and design was heavily motivated by incorporating the lessons from Rails' successes and failures.

What are the failures Phoenix addressed from rails?

Phoenix has less 'magic', for one. While it sometimes does mean there's a bit more boilerplate, I think it strikes a nice balance, and the 1.4 actually removed some magic.

Another one is the ORM, Ecto. It sticks much closer to SQL and I strongly prefer it over ActiveRecord. Ecto also includes the concept of changesets, which is a wonderful way to deal with validating user data and definitely an improvement over the Rails approach.

And finally, while I wouldn't necessary call it a 'failure' on Rails' part, the channels functionality in Phoenix is awesome for the more real-time apps that we have these days, and Phoenix is generally quite a bit faster in general than Rails.

> "I think Elixir has reached peak adoption"

I'd say the opposite. Core libraries have reached stability and engineering hours are switching to making easy what classic imperative/OOP paradigms have trouble doing:

- Dynamic web development without giant Javascript dependencies(Liveview, Drab, etc)

- Scalable distributed systems without giant teams (Firenest, Phoenix Channels/PubSub)

- Concurrency and data-infrastructure (Flow, Genstage, OTP)

- Reactive event-driven systems: OTP and the Actor model makes event-sourcing easier than you'll see anywhere else.

- Usual conveniences: User-management, HTTP clients/libraries

The next 2 years are going to get wild for Elixir.

For sure, and sockets are first class citizens and testing is at the forefront for all these things. Elixir is going to kill it.

And there's a lot of potential there for game backends as sort of a neat bonus. Just saying. : )

> a giant solve all your problems web framework

Phoenix's integration into my project consists of one file. Here it is: https://gist.github.com/tsutsu/03ea3036073994258ac1dee2e13a5...

Note that there's no MVC here, no i18n, etc.

Phoenix "is", fundamentally, just Phoenix.Endpoint, Phoenix.Router and Phoenix.Socket. In other words, it's a slightly fancier version of Plug that manages its own Cowboy instance and supports abstracted websocket message delivery.

Everything else is up to you; you can use whatever you like. The `mix phx.new` generates a rather "complete" skeleton, but that's just a suggestion; none of it is required to make Phoenix work.

The other stuff in that Phoenix skeleton isn't so much "a framework" (in the sense of e.g. Ruby on Rail's ActionPack, where you don't tend to see people using one of its components without the others, because they're all their own weird shape and only intuitively fit together with their "own kind.") Instead, the Phoenix skeleton stuff is just a set of common libraries that all get used regularly outside of Phoenix, and happen to each suit a need in web dev.

Or, to put that another way: Ruby on Rails is like GNOME (a bunch of libraries written by one group that—even if they are theoretically componentized and able to be used separately—nobody actually uses standalone if they're not making a "GNOME application"); Phoenix is like LXDE (a putting-together of a bunch of regular libraries that already existed to solve a problem.)

> The BEAM itself doesn’t fit well into the cattle model of reliability

One day I hope people will realize that a running BEAM process isn't a thing to put in a container; it is, itself, a hypervisor. You could have a Docker execution driver that deploys Erlang "containers" as relups, and that would make perfect semantic sense. (It just wouldn't be sandboxed in any way, so it's a security no-no for now.)

Picture a k8s cluster containing a mixture of Windows and Linux hypervisor nodes. A Windows-ABI container would get deployed to a Windows node; a Linux-ABI container would get deployed to a Linux node. The Windows "nodes" might actually be VMs running on Linux; the Linux "nodes" might actually be VMs running on Windows. Doesn't really matter, right?

Well, a BEAM-ABI container should get deployed to a BEAM node. That could, theoretically, be an http://erlangonxen.org/ instance; but it's probably a BEAM VM runninng on Linux. No difference, really. It's still a hypervisor addressable by the cluster.

> it has scalability issues when you reach a certain cluster size

The erldist protocol is made for distribution sets, not clusters. A distribution set is a fixed set of named nodes with named roles. Clustering is supposed to happen outside of the distribution protocol. (See WhatsApp's scalability talk where they talk about "meta-distribution" by connecting entire distribution-sets to one-another to form clusters of dist-sets.)

Effectively, ERTS is built to be used in RAID1+0 configurations: within each dist-set, you have a named master and a named hot-standby and some other named accessory nodes; and then you horizontally scale by partitioning your data across clones of this dist-set. (This is, specifically how Mnesia is meant to be used. You don't have a 1000-node Mnesia cluster; you have a cluster of 250 4-Mnesia-node distsets, with data routers in front.)

That said, you can ignore/replace erldist and just do clustering of ERTS nodes, if you like. Like in Lasp (https://lasp-lang.readme.io/).

> in my experience dynamic typing only works until you reach a certain level of scale

Erlang (and Elixir) were created specifically for writing the code that takes untyped data (e.g. binary packets) apart, assigning it temporarily into data structures so it can be analyzed and pattern matched, to try to figure out what its type should be.

There's no real way to get away from this, it's just an infinite regress. Use a typed RPC protocol like gRPC? You'll need to write the gRPC client/server and protobuf parser that your typed language uses. What language is best to write those parts in? Probably not one where everything has to already use static types. Parsing in e.g. C++ or Java is ugly. (Look at the internals of LLVM.) Erlang is honestly the best language I can think of to write those things in—not necessarily in the median case (Haskell's parser-combinators are simple if your use-case suits them), but in the general case, where the message grammar you need to parse being context-free or in any way well-thought-out is not guaranteed.

In my ideal world, there is a system architecture that consists of two languages—one that deals with untyped data and IO, and another one that deals with only statically-typed data, and is probably functional-with-exceptions. These two languages would be learned together. You'd write your code in an actor model, where the "outside" of an actor—the place where its ABI interfaces to the outside world—is defined in the untyped language; and then, once you've massaged the received message into the typed language, you'd write—inline—code in the typed language that handles that now-typed struct, doing something with it, and eventually returning a typed result from that sorta-like-asm{}-in-C scope. Then the untyped language would take over again, translating the response from the inner scope into a binary to go out through the actor's ABI.

I'm basically describing how Erlang works if you write the core of all your logic as NIFs in Rust/Haskell/etc. It's just clumsy, because there's no "first class" static language that Erlang embeds inline, rather making you build separate dynamic-library files using traditional tooling and then load them.

I like both Rust and Elixir, and I'm not sure I would use Rust for everything.

Case in point: I'm working on a fairly simple app - it runs continuously in the background, every minute downloads one file, and every second downloads a different file. These files are JSON, are parsed into domain structs, analyzed a bit, and occasionally persist to a DB.

I wrote this app in rust first, as a side project to better understand rust. The time I was seeing to parse a 50KB JSON file, turn parts of it into a map of structs, compare that map to the previously downloaded map, and log certain differences, was maybe 3-5ms, (compiled with --release).

I recently re-wrote that app in Elixir and the time it took to do that was... 3-5ms. If I were doing crazy calculations I'm sure rust would be faster, but I was actually surprised that in this real-world workload they were comparable in speed. And that's without dealing with a DB. Once you pull a DB in, the negligible processing time could be dwarfed by that anyway.

On top of that, the whole idea of starting up several processes to download each of the different JSON files on their own loop, and having it monitored in the supervision tree was a big win in Elixir, compared to spawning a couple threads and hoping they don't panic.

We replaced a little over 80k lines of Erlang with around 50k lines of Rust (not all at once of course). The biggest change was memory usage went way down. Erlang likes to run on nodes wth huge amounts of RAM for process mailboxes. We cut down our peak usage by almost 30%. That’s huge. Latency is also significantly better for the percentiles we’re looking at. The best part is that the code base is way easier to manage, and it’s much easier to keep adding to the code add without causing regressions.

Being a rewrite means that you understand the requirements well, which factor might play a big part in the performance story, regardless of language/platform.

> using a giant solve all your problems web framework (Phoenix)

Other people have said the same thing, but I want to echo the point that Phoenix doesn't need to be used as a large framework. I work on a large, production Elixir app, and all of our business logic is implemented as separate apps that don't even include Phoenix as a dependency. They're just structs, modules, functions, and supervision trees.

We then depend on those apps from our various frontends (web, api), which is where the Phoenix integration happens, and at that point it's more or less just template rendering and routing.

So you are just going to do this now in every Erlang/Elixir thread?

Rust sucks especially for the problems that Elixir is good at. Maybe when writing some system tools I'd consider Rust but so far it has been an absolute pain to use. Can't even get the code to compile for hours while in Elixir I've gone and done it in a flash and there have been almost 0 errors. Almost like Haskell's "if it compiles, it works". No need to attack Elixir with Rust everywhere.

When working on V2 of our architecture we tried out both Elixir and Go extensively. For our workload Go was roughly 8 times faster than Elixir, while both handling concurrent workloads exceptionally well. (bit of background on the type of work we do: https://stackshare.io/stream/stream-and-go-news-feeds-for-ov...)

That being said, I'd love to find an excuse to work on a little project with Elixir and Phoenix. Some really cool concepts. Especially the build in support for realtime is really nifty. With Go that's much more work to get up and running.

I can't say I'm an expert in go - I only wrote the cli for my program in go, but I feel like elixir optimized for developer productivity far more than go does. Most of the time you don't really need that level of performance.

You're going to find that for workloads that require churning through requests, or something similar, that Go or Rust or something similar is simply better. Elixir/Erlang is mostly good for dealing with a lot of persistent state, staying up and running, and reliably getting data from point A to point B. But none of those it does particularly fast, unless you can do it without needing to mutate a data structure/variable, or you can take advantage of the fast binary pattern matching.

> With Go that's much more work to get up and running.

Can you expand on that? Go is very fast to get up and running, you DL go runtime, git clone a repo, go build . and have your single executable ready to go.

This bit of Phoenix: https://phoenixframework.org/blog/the-road-to-2-million-webs... Takes substantially more work in Go.

Thanks for the answer, for Go it doesn't take much most likely easier than in Erlang: http://goroutines.com/10m

I never had to work on that many connections per instance, but knowing how much faster Go is vs Erlang I woudn't see this as an issue?

He's referring to Go being more work to setup realtime web things, as opposed to Phoenix.

> consider going old school and rendering your HTML on the server; you might be delighted

One idea I'd wish more sites would consider.

Yeah, writing about using Elixir at The Outline is cool and all, but let's not forget this place wiped out most of its staff, including developers.

Care to point us to the story?

I would like to use Elixir more at work, I'm hoping that we'll be able to use it for some new projects. I love the language and the community.

Can you print an example of the random test error? I run a decent sized phoenix project (150K lines) and likely can help you out if you give a bit more detail.

With regard to associations, for simple belongs_to associations I will usually write separate changeset functions where the first argument is pattern matched against the record. I will give an example with Post/Comment

    defmodule MyApp.Blog.Comment do
      def changeset(%Post{} = record, params) do
        record
        |> build_assoc(:comments)
        |> changeset(params)
      end

      def changeset(%Comment{} = record, params) do
        record
        |> cast(params, [:body])
        |> validate_required([:body])
        # Continue with validations, constraints, etc.
      end
    end

In this case the changeset function calls itself once it handles building the association. For more complex conditions, I will usually write a "Form" module that may handle that case usually with a Enum.reduce combining multiple calls to `put_assoc` so that I can make nested associations in one insert.

Hope this helps a bit.

>>def changeset(%Comment = record, params) do

Small typo: should be %Comment{}

thanks, fixed

Does anyone miss or want type checking when working with Elixir? After working with Ruby to build large systems for so long, I'm missing a little bit of the safety that types provide. Typescript has been great.

Yes and no. I also come from a similar background with the pain of building a large system in Ruby, and having been doing Elixir for 2 years since then I find it that is has the right tools to let you go either way to a degree.

I like that you have all sorts of optional tools for adding runtime type checks like guards and struct pattern matching if you want it. Using it extensively will inhibit generalizability, but it can be tremendously useful for isolating errors in problematic code paths by failing early and explicitly. Typespecs enforced by Diayzler aren't exactly a safe type system, but can help with some of the sanity checks, documentation, and linting that Typescript gives you (not really being a safe type system either).

I would say the only thing I really feel is missing is being able to check for protocol implementations and perform runtime control flow based on these.

Gradualizer supports Elixir. It will move into beta soonish :)

I come from Scala to Elixir and I personally prefer static typing for a lot of reasons. I have come to very much enjoy the looser typing as I can write more code more quickly, serialization is less of a pain, etc. Elixir is still compiled (and you have dialyzer if you really want to go that way.) You can match on a struct type and the compiler will catch a typo if you try to access a member that doesn't exist in the struct so you can go a long way to lean on the compiler. All that said, I can attribute +50% of the bugs that hit production to things that a type system would have caught. I would prefer to go back to writing code with Scala/Akka and would choose those technologies in a blink for any system that I'm going to grow over time. development time _is_ slower with scala/akka (assuming you don't count time spent trying to refactor or fixing bugs a typesystem would catch) but it's probably an order of magnitude faster for computational tasks, not slower on concurrency related items, has a bigger/more mature eco system (which is huuuuge) and gives you OO beside FP which is highly valuable for design. The skill ceiling is higher with scala/akka so you can keep growing and learning.

And OMG trying to refactor Scala vs trying to refactor elixir?? IntelliJ is a jet powered pogostick for applying refactorings (to quote Beck) - It's very had to make large structural changes to elixir code without a static type system and tools to help you rename, extract methods and classes, and move them around. I can do things in scala I would never dream attempting without aa type system which is probably my biggest gripe for long term maintenance.

That being said, working with elixir has turned me into an Emacs junkie and I personally find a lot of value in that but it doesn't help me deliver better code faster, just gives me nerd points because I can check the weather and reply to email from my textt editor

Elixir has a specific kind of typechecking: success typing. You can set up the Erlang Dialyzer tool to work with VSCode using the ElixirLS extension. It will warn you about type errors. Success typing means that there is provably a type error and you actually need to fix it, as opposed to the more common form of typechecking wherein the compiler forces you to prove that there isn't a type error.

True to form, when I see a Dialyzer warning and think about it for a bit, I find that I always need to fix the code. The error messages do take some getting used to, though.

Isn't Dialyzer's success typing similar to TypeScript in that it's gradual typing? Or am I muddling up my concepts?

It's similar and they are both considered gradual typing, yes. Dialyzer is 'better' in some ways though. For example, consider this TypeScript:

    function f(a) { console.log(a.x); }
    f({});

With the strictest checks turned on ( try it at https://www.typescriptlang.org/play/#src=function%20f(a)%20%... ), TypeScript will give you this message: `Parameter 'a' implicitly has an 'any' type.`

Now consider the equivalent Elixir:

    def test2(), do: f(%{})
    def f(a), do: IO.inspect(a.x)

Dialyzer will give you this message: `Function test2/0 has no local return ... The call 'YourModule':f(#{}) will never return since it differs in the 1st argument from the success typing arguments: (atom() | #{'x':=_, _=>_})`

This is why I said the Dialyzer message takes some getting used to :-) But, it just means that `f` needs to be called with either an atom (because you could pass in a module to the function, and modules are modelled as atoms), or with a map object with the `x` key and possibly some other keys.

> 90ms is a 90th percentile response time on The Outline

How much of that is the DB, and what DB are you using?

I like the BEAM environment quite a bit, myself, and have used it successfully in several semi-embedded environments, where it really shines because of its robustness and reliability.

Postgres.

Currently, our mean response time for the route with the most throughput is 16ms. Its to measure exactly how much of that time is being spent in the database, but the main query takes about ~6ms.

- Ecto is still a climbing hill for me. In fact I'm currently trying to warp my head around changesets and associations.

- Same for GenStage. I wanted to use them for workers pulling from an episodic queue but the docs with the emphasis on consumers pulling multiple events at once from a never ending stream didn't really speak to me. Tailoring to my usage looked verbose: storing demand etc. I'll have to revisit when I'm better at Elixir.

- I still think caching should be part of Ecto in some way, at least in the docs.

- I would still like more straightforward template inheritance in Phoenix (think Jinja2, Twig).

Excited for the future releases of Phoenix 1.4 and Ecto 3.

I'm not sure exactly what made Ecto click for me, but once it did, it quickly became my favorite ORM/data mapper.

Have you read some of the Plataformatec blog posts about Ecto?

I thought similar but once Ecto clicked for me, everything else felt clunky. I don't know what was the exact cause but using it for a real project helped.

Also Programming Ecto is great.

https://pragprog.com/book/wmecto/programming-ecto

Changesets are great. I was hesitant when I first used them coming from Rails, but they're very useful. Combined with the contexts, they help remove the need for implementing ActiveRecord callbacks; which caused a lot of problems with Rails apps as they aged.

I had a hard time with those too! However, changesets are really awesome. Easiest, bastardized intro is to think of it as preventing a trip to the database/data sanitization before doing any sort of state modification. Then if you stay open minded later you will learn other things that it can do as you along.

A changeset is basically an accumulator object that contains both a diff and DB operation specifics. It basically says "I have to change this object's 3 fields to the following values and then perform a DB update". It can also contain validation errors. And... that's all there is to it. And you can use it without a database. It's perfectly useable when you accept external data and want to inspect how would it modify your state. No persistence need be involved.

For everything else, poke us at ElixirForum. People are helpful there.

You should definitely CDN your data. A majority of a client-side latency is spent round-tripping to the server. Using a CDN greatly reduces this time.

I don't know your workload, but 60ms doesn't seem that fast. Don't get me wrong, it's pretty good, but for a read-heavy load it seems like this could be optimized to be sub-10 ms at the 99th percentile.

90ms is the mean across all routes. Our post route, with the highest throughput, has 16ms request/response time. We do use a CDN, and we spend a lot of time thinking about performance. Thanks for your insight.

An app I maintained that was doing anywhere from 3 to 50 DB queries per request, 98% of the time was spent in the DB roundtrips. The Elixir code itself took 0.2ms to 3ms.

It's really lightweight. Can't be anywhere near Go or Rust of course but it's provably much faster than Ruby, Python and PHP, and very often outperforms JS on V8.

Most of the overhead I observed in Elixir apps stems either from external network requests (DBs, K/V stores etc.) or CDNs / static assets. Most of the Elixir code is almost invisible in terms of footprint.

> it's provably much faster than Ruby, Python, PHP

This is simply not true. Ruby + Sinatra/Roda/Hanami + Sequel even performs better than Elixir + Phoenix + Ecto in some cases. It's basically dead-even. For example: https://www.techempower.com/benchmarks/#section=data-r16&hw=...

BEAM has the benefit of being a register-based VM vs the stack based VM of CRuby but the immutable semantics of Erlang/Elixir negate a lot of the VM and GC advantages.

You should read a bit more on how the various popular benchmarks manage their different code bases and not just quickly believe the graphs. There's a lot of story behind them.

Several languages, Elixir included, are misrepresented due to badly written and non-idiomatic code (caveat: this was true a while ago, I haven't checked lately though). And when its users try and open a PR they are shot down with "We don't allow cheats". No cheats though, just very normal language best practices.

So I disagree with you here. I maintained a ton of Rails and several Sinatra apps and to say they are on the level of Elixir makes me wonder what kinds of hosting budgets we both had access to and were they really so hugely different.

(Sinatra was actually very decent but it being Ruby it doesn't scale well and Erlang/Elixir work much more stably until they start having problems compared to it.)

Again, don't just trust the popular benchmarking suites. You will quickly discover they usually either have an agenda (like the Crystal language creators always showing how quicker it is compared to Elixir, and refuses any Elixir PRs -- seriously no idea what the Elixir community ever did to them) or only pay lip service to representing all languages fairly (like TechEmpower does).

(Edited because I wrote the first version after a nap and it came off sharper than I intended.)

There is an implication here that we (TechEmpower) have rejected PRs that would make the code in our Elixir tests more idiomatic. And if I understand you correctly, when we did so, we explained that the implementation did not comply with the spirit of our tests. To be clear, we do reject pull requests that we believe are cheating (e.g., caching results or other means to avoid specific workloads called out in the test requirements).

Can you point to which Elixir-related PR [1] we rejected on that basis?

Some of our most recent Elixir commits were contributed by a gentleman named Michał Muskała. For example, here are some he characterized as "minor optimizations" [2]. Since he is a one of 16 people on the Elixir language organization page [3], I take his contributions as representative of idiomatic code or at least what the organization wants to represent as idiomatic. Do you have a reason to disagree?

[1] https://github.com/TechEmpower/FrameworkBenchmarks/search?q=...

[2] https://github.com/TechEmpower/FrameworkBenchmarks/pull/3848

[3] https://github.com/orgs/elixir-lang/people

Thanks for chiming in.

Michał Muskała is a core contributor and his opinion weighs a lot so I am really glad you guys are working with him.

I did check out the PR and his changes seem to be on par with what I've seen in other repositories used in TechEmpower's benchmark. Not cheating, just making sure not to use extra baggage that can severely hamper a framework's performance.

As for rejected PRs, I looked but couldn't find what I roughly had at the back of my head. It was a while ago and since I can't find it now I will have to concede that I remember incorrectly. (Ouch.)

As I pointed out, my info was outdated. I see things have improved in the meantime, which is good news!

Thank you for being responsive and transparent. Much appreciated.

There are plenty of other benchmarks where Ruby on YARV and Elixir on BEAM are roughly even. In this example, JRuby is ~2x faster than Elixir + BEAM.

https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac...

As for CRuby and JRuby, they are completely different runtimes. Not sure they belong in a comparison. Elixir only has one runtime, let's compare with the defaults.

Additionally, why do you use the scripting mode of Elixir (.exs file extension)? Why not compile it (.ex file) and run it? The scripting mode hampers performance. Can you update the gist with a compiled version?

And finally, comparing raw performance isn't really what we argued about here (at least I wasn't). I thought we talked about I/O and concurrency capped environments which are the conditions in which the usual web / API app lives.

Erlang/Elixir shine on a lot of things. But definitely not on raw crunching power -- that much is true and the Elixir community is realistic about it.

I actually even tried compiling the module with HiPE but it made almost no difference so I didn't update the Gist. I've updated it to reflect the tiny improvement it made.

In terms of web, CRuby 1.9+ simply uses OS threads for concurrency and parallel IO. You will run out of CPU resources before "the runtime won't scale" on modern Ruby.

Reply on news.ycombinator.com