Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?
When you're coming from Rails/Sidekiq world a process can seem very fragile because you're usually thinking of it like a thread or a Sidekiq job.
With a Sidekiq job, the jobs are kept in Redis and what's actively being worked on is tracked by Sidekiq in Redis. Even if you switch to a different store like Mongo for job details, you still have to have Redis available just for the progress tracking.
This background job will usually run through and execute a lot of sequential steps. When writing a Sidekiq job you are supposed to build it so that if it fails at some point along the way, the job can run again. Failures will trigger retries or exhaustion failure cases and log the error out to Redis for your Sidekiq dashboard so that you can do something different with it at that point.
The job is running in it's own thread which doesn't have a clean tracking mechanism via Sidekiq, so the only option to retry happens by re-queuing the job. Your database connection pool has to be increase to match your concurrency level in most cases as well and the concurrency has to be closely watched to track resource usage by job type in multiple job queues to ensure that too many heavy jobs don't take over and prevent other things like emails or push notifications from getting out on time.
That's with everything that's built in plus a few extensions.
Now, compare that to how processes work with Elixir/Erlang.
Processes are so small (0.5kb compared with at least 1mb / thread) that each process is created with a supervisor as a standard mode of operation. Each job has it's own personal supervisor that will see if an error happened and immediately restart the job (with backoff). No external dependency, re-queuing or special tracking needed (although there are plenty of options).
Because the processes are so small, in most cases you won't have a single large background job, you'll have many smaller ones breaking the entire thing into smaller pieces with other supervisors so at the point of an error, only that piece will need to restart.
Worrying about different job sizes or resource usage largely goes out the window as well because the BEAM's scheduler limits the execution time of every process ensuring that a single large job can't takeover a machine. All of the smaller ones will keep executing just as if the big one wasn't running.
Because the entire thing is built for concurrency, you also don't have to worry about the database-connection-per-job as the connection will just be checked out to make a query and then checked right back in. Because the scheduler could change context at any point, allowing a single process to hold a connection for longer than the database transaction in progress doesn't make sense.
If you really want to use the Sidekiq approach, there is a library for it available (https://github.com/akira/exq). But what you get out of the box with Elixir/BEAM is already pretty far ahead of what Sidekiq gives you.
I'm definitely open for new ways because the whole Sidekiq approach does work but it's a very brittle system. Not because it's a bad tool, but there's just so many moving parts and a million things that need to happen at many layers for it to all work.
Do you happen to have a blog post or code sample that ties together creating a production-ready "Elixir job"? Out in the wild it's reasonable to expect to have code fail for reasons you can't control and being able to control what happens when that does or how it re-tries is important.
A big part of the difference it that in a typical web application in most other languages, you'll think of your system in two parts: web and background.
With an Elixir/Erlang you tend to think of the web as just one interface to the application itself. The idea of background jobs in those contexts somewhat goes out the window when you have an environment where spinning up millions of processes at the same time is the expectation and not a terrifying scaling scenario.
Think of responding to an incoming web request as just another job, essentially.
Because everything is just message passing and there is no global state, the restart just looks like a function call. The supervisor will start a new process calling the function with the arguments that were passed to it when it was created.
Here's an example that will try to explain things.
https://medium.com/@tylerpachal/let-it-crash-creating-an-exa...
Also, I discovered your blog. Good stuff!
Currently reading your Insanity with Elixir + Phoenix post at http://www.brightball.com/articles/insanity-with-elixir-phoe....