Comment by joelbluminator | Hacker News Reader

Comment by joelbluminator | Hacker News Reader

Parent Full thread

Sidekiq is not a workaround. This is a myth. Most companies need to persist their jobs (what happens during deployments or when a process dies? do u just let all the job info disappear?) and have some kind of queue system. Even Elixir has some redis backed queues. BEAM, OTP, concurrency, this whole thing is solved with the current fashion of devops teams and kubernetes. It doesn't really matter what tech stack you use anymore it can quite easily scale. Our devops team is 2-3 people and they're scaling our Rails architecture easily. Our scale is quite big since we're a b2c company and most of their effort isn't even on scale but on making deployments easier and troubleshooting all kinds of dev problems / pager duty alerts. They would do the same amount of work if they were doing that with Elixir. If you're hitting Whatsapp scale yes Ruby is less than ideal. Could we stop pretending like the challenges of 5 companies are what most devs need to go through?

In Elixir you'd use Oban instead of sidkiq and you get more performance and it scales out horizontally with your app. Each new instance of your app is essentially a new sideqik server as a bonus

edit: > BEAM, OTP, concurrency, this whole thing is solved with the current fashion of devops teams and kubernetes.

This is a hilarious statement which I hope is satire

I don't know about Oban. I don't think it's bad design for the workers to be separate from your main app like in Sidekiq (they do 'require' the app but they are essentially separate from the servers). Anyway making this point as a huge win for Elixir over Ruby seems really exaggerated to me.

It prevents unnecessarily moving data around. And if you're running the whole stack on the same server with Sideqik you're dealing with interrupts and memory copies between processes; in Elixir it's all within the same allocated memory and nothing gets copied. Plus it's another service you don't have to monitor because it's automatically monitored by BEAM.

What if you want to scale your job workers and your servers separately?

I think it's nice that you can run everything in the same server. If you're scaling, and want to split everything up, that's a "nice problem to have", but it's convenient if you can just run everything in the same application/codebase when you're getting started.

I do agree with you that's not a huge win compared with Rails, but it is nice to have. I think you'd have to look more at something like "lots of concurrent, long-lived connections" for the real wins over Rails for the BEAM ecosystem. I mean, you can do that in Ruby if you want to, but it's going to be cleaner and simpler with Elixir/Erlang.

Oban will let you choose which servers the jobs can execute on, and you could do the same with your own app code

> Queue Control — Queues can be started, stopped, paused, resumed and scaled independently at runtime locally or across all running nodes (even in environments like Heroku, without distributed Erlang).

I don't see BEAM and OTP as being synonymous with Kubernetes. They have some overlaps but they aren't drop-in replacements for each other. If anything, they are complimentary technologies.

The advantages of BEAM and OTP are that I can spawn new concurrent processes throughout a cluster and then use the actor model to send direct messages to those processes from any other processes, regardless of where the sender and receiver happen to be. I can also easily configure Behaviours so they automatically run on either a single node or every node in the cluster, depending on what they are doing.

In both cases, if a node goes down, Elixir will automatically restart any affected processes on another node. When a process has an issue, I can surgically handle that issue at the individual process level. Kubernetes doesn't handle problems that granularly.

One advantage of Kubernetes is autoscaling. OTP and BEAM don't do that.

By using libcluster, you can combine Kubernetes and BEAM/OTP and get the best of both worlds.

Reply on news.ycombinator.com