Comment by berkes | Hacker News Reader

Comment by berkes | Hacker News Reader

I agree with most of the findings in that article, but I miss one very important one: consistency.

Jobs in your queue in the database can be committed in the same transaction that makes the mutation. With redis you have no such thing. Leading to (been there, it hurt) potentially missing jobs or queuing jobs that have no associated mutation in the db.

Furthermore, a big downside of redis is that it's promise of consistency is low. Mostly by design: redis is not meant to be the primary, one and only store of truth. Commonly, people store stuff in redis that can be re-built from database or some eventlog or so. But not sidekiq. If your redis starts failing (been there, it hurts), e.g. though OOM errors, you are loosing data. Jobs, with their parameters, are dropped, often irrecoverably. And with Rails' common set-up, this means you are irrecoverably loosing data: there simply is no way to find what emails you did not send, nor any way to re-queue them. There is no way to re-enqueue those "generateReport" jobs using the data at that point in time and so on.

Basically: Redis is quite certainly not the best tool to store your job-queues with associated data, in. Sidekiq is the less sturdy approach of both.

Transactional consistency is nice. Maybe worth it for some low-volume jobs.

The downside that comes with it is using your primary DB as your queue is that the queues in an app tend to absorb all the exceptional conditions (queue grows to millions during worker outage, backfill which needs to process millions of items). Using your primary DB means all that disk usage, memory usage, and IO hit your primary DB in exceptional ways and impacts your app in more ways than just queue delays. It also scales much less linearly than other queue options.

A nice middle ground is to store all your data and state in your DB transactionally, and only queue tiny lightweight jobs with database primary keys (database: sentWelcomeMailDate=nil, queue: sentWelcomeMailForUser:1234).

This is another case for neither DJ or Sidekiq. A fully hosted option like SQS addresses the durability concerns of Sidekiq, and the scaling concerns of both. I’ve queued 1B+ jobs in SQS and watched the workers slowly burn through them over the course of a month.

DJ and Sidekiq can go quite far. I’ve used both and patched some perf issues on DJ. The lesson learned was queues at scale are hard distributed systems problems - you probably don’t want to have to know how they work, or host the distributed system yourself.

> Maybe worth it for some low-volume jobs. > ... > The lesson learned was queues at scale are hard distributed systems problems

I'd go so far to say that when you need such scale, Rails or Job Queues like Rails' are not the solution. Event-based architectures (event-sourced, event-buses, actors etc) are far better suited.

Rather than trying to solve all the problems that those architectures were explicitly designed for in redis/sidekiq/rails, just move to those architectures.

You won't have the nice "rails generate CreditCard" anymore /s, but you'll gain a lot of consistency, simplicity, performance and architectural sanity.

Totally. Or use rails for your web app, and something else for your queue processing. Rails is nice for rails things. Rails is scales for serving web apps much better than DJ/Sidekiq do at serving queues.

Our solution was to build middleware/interfaces that only push jobs after a transaction has committed. There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.

Agreed on the Redis health being critical and if it's unhealthy you can't trust anything. We over-provision compute and memory and enable persistence as a result. We are also mindful of our payload sizes, which we also compress. Additionally any jobs scheduled beyond a certain time threshold are committed to the DB, not Redis, and we have a periodic job that adds them back to Redis as we approach their scheduled time to run. This is good for both durability and keeping the surface area of Redis work smaller.

It's not perfect and there is probably a day where we switch it out but for now we have developed a deep understanding of Sidekiq and Redis and get to leverage the larger Sidekiq ecosystem. This has worked well for us to the tune of about 35 million jobs a day.

> There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.

If it happens though is it a big deal? I’ve seen Sidekiq jobs used to move money, for instance. This can result is Bad Times. Right tool for the job, if consistency is a requirement Sidekiq I think gets incorrectly applied in a lot of applications.

I don't move money, but if I did I would do it like we do for most objects in our system (like interacting with S3 objects for example), I'd represent that transaction (in the money sense) in Postgres with a state and the job would be expected to do work and update the state. The transacting job itself could be queued immediately for convenience but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency.

In this sense the job acts on data, it doesn't hold data, and you can build in backstops in the event of any kind of failure, not just Redis. You'd want to build this into your architecture regardless of what your background job tooling and infrastructure you use.

We worked on the same problem, and what you described works, however it also happen to be exactly what delayed job does.

Which bring me to the next point: Sidekiq should never be the default choice. It brings distributed systems complexity to an audience that's trained to work on a monolithic database.

Delayed job or similar should be the default, with sidekiq relegated tp jobs ready to be lost

I agree with you that people often overlook the complexity of tying changes to their database, with enqueuing jobs in a completely separate system.

https://www.betterment.com/engineering/delayed-resilient-bac... does a great job explaining this.

> but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency.

You've re-implemented a job queue in the database.

No I didn't.

You kinda did though. Your models in states they shouldn’t be are jobs. These kind of headaches are avoided using a job queue that provides strong consistency.

Everything is a trade off.

You can run Redis in a persistent setup. So each Redis instance has a replica and both master and replica write out to disk. You still don’t get transactions, but depending on your scale a transaction for every job you enqueue may be impossible.

If you’re small enough that you don’t knock over your DB by putting jobs in it then the transactions are nice!

As you say, looks like redis-raft is capable of solving durability and consistency in a replicated environment as of 2020, which is welcome news: https://jepsen.io/analyses/redis-raft-1b3fbf6

At Betterment we use our OSS mostly-compatible fork of Delayed::Job referenced elsewhere in the comments to enqueue and work millions of jobs a day and sleep much better at night with the at-least-once delivery semantics if-and-only-if the related transaction commits which you can’t get without some form of integration with your primary database.

With listen and notify in Postgres, it's a fantastic choice for a job queue if you're not crazy scale (which is most apps, honestly).

Especially with transactions as mentioned, with elixir and ecto and Oban, you can ensure your jobs are inserted properly. And it's easily testable.

I have strong opinions on this, but IMO if you are thinking about using PG pub/sub, you'd almost always be better served by just running a worker that reads from a logical replication slot. Little bit heavier lift, but there's plenty of reference code out there, and you get consistency + delivery guarantees.

Reply on news.ycombinator.com