Verk – An Elixir job processing system backed by Redis | Hacker News Reader

Verk – An Elixir job processing system backed by Redis

github.com

39 pointsgeordidearns10 comments

I don't understand why we're rebuilding Resqueue/Sidekiq in Elixir. This seems like a problem which is solved largely by OTP.

Yes there are issues with back-pressure and over-filling mailboxes, but I can't help but wonder why you'd want to bring Redis into the picture when asynchronous processing is built-in for you.

Similarly if you want more robust queueing semantics, isn't RabbitMQ there?

Am I missing something?

OTP won't out of the box persist jobs that you want to guarantee that it will run. It won't also retry and it can't schedule jobs to happen in the future.

Unless you are willing to lose the jobs that are just in memory you can simply use Task and send messages.

RabbitMQ also does not handle retries with backoff, scheduled jobs, etc.

Disclaimer: maintainer here.

Edit: I added the "with backoff" to not mix with automatic reenqueue if you reject a message from a Rabbitmq consumer.

Fair point. Why then persist to redis, rather than using something like mnesia and keeping it all in Elixir?

Is there a compelling reason to add an additional piece of tech in my stack?

That's a great question. I chose Redis for 2 reasons:

* Simple to use

* Compatibility with other systems like Sidekiq/Resque. Out of the box it will understand their clients.

My experience with Mnesia is close to none but I know it's not trivial to deal with split brain scenarios for example. It's also complicated to connect multiple nodes for my use case (using docker) and also to deal with moving instances inside a cluster. I would need to care about the locality of the data.

One can use any Redis service (AWS Elasticache, Redis Labs) and not care about maintaining a database. The Verk user just needs to have a working Redis database.

I'm pretty sure that having the database in the same space as the job processing system would speed up things. I would love to see a project that tried this approach. Probably not using JSON but simply Erlang terms.

Seems reasonable enough. I just feel like shouting from the rooftops "redis is not a queue server" but you already know that :)

Honest question, what is redis then? I've used it for all sorts of reasons, it seams like a swiss army knife, caching, queing, pub/sub, persistence, non-persistence all-in-one utensil. You can do a lot with it, and if it's already in the stack, it's a lot easier for others to reason with.

> there are issues with back-pressure and over-filling mailboxes

Have you seen ferd's pobox library [1]?

[1]: https://github.com/ferd/pobox

Why not https://hex.pm/packages/exq ?

Here's a detailed explanation: https://github.com/edgurgel/verk/issues/2

why Redis when Elixir supports the Erlang native ETS and DETS libraries?

Reply on news.ycombinator.com