Hacker News Reader

Comment by nickjj | Hacker News Reader

> Forget using a million different technologies for things like background jobs, OTP can supply you with everything.

It's true but in practice this usually doesn't pan out.

For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app.

Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.

View on HN

I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points:

Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time

Tracking failures/sucesses: postgres

Exponential backoff: a function

Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2)

draining: remote console

periodic tasks: a simple library that knows how to cron

I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff.

Erlang has plenty of drawbacks, but this isn't it.

> There's usually (almost always) durable storage required somewhere.

Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.

Despite? It pretty much means "to remember", it doesn't have the alpha privative of amnesia, "to forget"

IIRC original name was Amnesia, but Joe was made to change it for marketing reasons (it's in one of his books or blog posts).

Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.

At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine.

The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.

(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)

I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think.

I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year.

Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :)

(Hmmm, now I'm trying to figure out you are, send me an email if you will :)

Mnesia is first and foremost and in-memory key-value configuration storage. Relying on it for anything else will damage you in the long run.

> I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert.

Which is why I'm happy others have paved the way.

A huge amount of web apps can get by with a Postgres backed queue that won't break a sweat handling a few hundred writes per second on a low end cloud hosted VPS.

While leveraging something like Oban you don't even need to be an expert. You can just use the library, throw work into it with a reasonable amount of foresight / best practices and you can start developing features in your app.

Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only.

We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).

A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.

Phoenix ships with Ecto ORM-style-binding to Postgres underneath (usually) for a good reason! :) And if you need that in memory magic where the serial nature of a genserver won't suffice for your load, you basically have built-in Redis with Erlang's ets. And if you really need insane performance on longer-cached items than ets is optimal for, Discord released https://github.com/discord/fastglobal

And if you don't want to put all those pieces together, like you said there's a rich set of modules people have built, like oban. Long rise and live the elixir monoliths ;) (although admittedly even Discord had to swap out some of their most intense functions with Rust, but how many people are writing systems that need to handle millions of concurrents?)

> you basically have built-in Redis with Erlang's ets

The big difference there is if you have a Dockerized app, Redis won't be restart on every deploy but the BEAM will be so you will lose your ETS cache or whatever you're storing.

Now, I know cache is meant to be disposable but this model of using ETS instead of Redis reminds me of a Python or Ruby app storing its cache in memory. I mean, sure you can do it but for the longest time (7-8 years?) the community as a whole kind of landed on keeping that stuff out of your app's process.

This is my experience as well, working with Elixir fulltime for about 5 years.

Reply on news.ycombinator.com