Hacker News Reader

Comment by davidw | Hacker News Reader

Business-wise, what do you think the sweet spots are for LiveView, Phoenix and Elixir right now? I love the BEAM ecosystem (I've been using it since 2004, on and off), but for a lot of places, Rails is still a great place to start. What kinds of applications have you seen where all the BEAM features just make Phoenix et al not just a little snappier, but a clear winner?

View on HN

The sweet spots for me are:

1. It is much easier to trace things through the entire Phoenix stack than it is in Rails. It is also much easier to add things to the Phoenix stack using plugs.

2. Elixir is concurrent, whereas Ruby is not, so when performing long-running processes, you can just do them in Elixir/Phoenix without having to rely on workarounds like Sidekiq, Resque, RabbitMQ, etc.

3. Writing multi-threaded applications is much easier in Elixir than many (if not all) OO languages.

4. Pattern-matching for variables and functions, and binary pattern-matching for parsing text.

5. Mix.

6. BEAM, OTP, and supervision trees.

Looking more at things like #2. Sidekiq isn't terrible though, and if you get big enough, you're going to want some more management than just spawning a BEAM process to do something. What have people settled on for that with Elixir/Erlang?

You could use something like oban: https://github.com/sorentwo/oban

Sidekiq is great, but it is a thing you have to think about (in particular, how you're shuttling state from workers to a place where front-end can see it). The big win with Phoenix as I understand is not having to think about it at all: it has a natural expression in the language and the platform, and that expression is performant.

If it's just a small fire and forget thing in Erlang, you can just spawn the process, sure. My point is that once people start to care about what happens with those background jobs, you probably need some infrastructure around them, and then the Erlang thing might start to resemble Sidekiq a bit more.

This.

"When we want to have more control, i.e. persistence on disk, a workers pool, retries, among other things, there are several popular solutions:

Oban: a robust PostgreSQL-based queue backend process system. It has a UI in its pro version Exq: a Redis-based Resque and Sidekiq compliant processing library. It has an open source UI exq_ui verk: same as Exq. Is is compatible with Resque and Sidekiq job definitions"

https://sipsandbits.com/2020/08/07/do-we-need-background-job...

So basically sounds to me like Elixir is doing pretty much the same thing as Ruby. We usually do want more control and disk persistence, so...

Sidekiq is not a workaround. This is a myth. Most companies need to persist their jobs (what happens during deployments or when a process dies? do u just let all the job info disappear?) and have some kind of queue system. Even Elixir has some redis backed queues. BEAM, OTP, concurrency, this whole thing is solved with the current fashion of devops teams and kubernetes. It doesn't really matter what tech stack you use anymore it can quite easily scale. Our devops team is 2-3 people and they're scaling our Rails architecture easily. Our scale is quite big since we're a b2c company and most of their effort isn't even on scale but on making deployments easier and troubleshooting all kinds of dev problems / pager duty alerts. They would do the same amount of work if they were doing that with Elixir. If you're hitting Whatsapp scale yes Ruby is less than ideal. Could we stop pretending like the challenges of 5 companies are what most devs need to go through?

In Elixir you'd use Oban instead of sidkiq and you get more performance and it scales out horizontally with your app. Each new instance of your app is essentially a new sideqik server as a bonus

edit: > BEAM, OTP, concurrency, this whole thing is solved with the current fashion of devops teams and kubernetes.

This is a hilarious statement which I hope is satire

I don't know about Oban. I don't think it's bad design for the workers to be separate from your main app like in Sidekiq (they do 'require' the app but they are essentially separate from the servers). Anyway making this point as a huge win for Elixir over Ruby seems really exaggerated to me.

It prevents unnecessarily moving data around. And if you're running the whole stack on the same server with Sideqik you're dealing with interrupts and memory copies between processes; in Elixir it's all within the same allocated memory and nothing gets copied. Plus it's another service you don't have to monitor because it's automatically monitored by BEAM.

What if you want to scale your job workers and your servers separately?

I think it's nice that you can run everything in the same server. If you're scaling, and want to split everything up, that's a "nice problem to have", but it's convenient if you can just run everything in the same application/codebase when you're getting started.

I do agree with you that's not a huge win compared with Rails, but it is nice to have. I think you'd have to look more at something like "lots of concurrent, long-lived connections" for the real wins over Rails for the BEAM ecosystem. I mean, you can do that in Ruby if you want to, but it's going to be cleaner and simpler with Elixir/Erlang.

Oban will let you choose which servers the jobs can execute on, and you could do the same with your own app code

> Queue Control — Queues can be started, stopped, paused, resumed and scaled independently at runtime locally or across all running nodes (even in environments like Heroku, without distributed Erlang).

I don't see BEAM and OTP as being synonymous with Kubernetes. They have some overlaps but they aren't drop-in replacements for each other. If anything, they are complimentary technologies.

The advantages of BEAM and OTP are that I can spawn new concurrent processes throughout a cluster and then use the actor model to send direct messages to those processes from any other processes, regardless of where the sender and receiver happen to be. I can also easily configure Behaviours so they automatically run on either a single node or every node in the cluster, depending on what they are doing.

In both cases, if a node goes down, Elixir will automatically restart any affected processes on another node. When a process has an issue, I can surgically handle that issue at the individual process level. Kubernetes doesn't handle problems that granularly.

One advantage of Kubernetes is autoscaling. OTP and BEAM don't do that.

By using libcluster, you can combine Kubernetes and BEAM/OTP and get the best of both worlds.

Reply on news.ycombinator.com