Resque Bus

I preferred the approach described here over pure Redis pub/sub because of the ability to disconnect or lose connections and still end up with the data eventually.

Specifically about "cron" capability, resque-scheduler does a pretty good job for Resque in general. Resque Bus uses it for exponential-backoff retry.

It uses the scheduling capability for the "Heartbeat" described in the README. I'll write more about that later if people are interested. Quickly, it's an automated way to add events every minute. We use this like a global (shared across applications and servers) cron clock.

Sidekiq supports scheduled jobs: https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs

>Currently all redis-based background job implementations does not support scheduled jobs.

That has been false for 4 years.

https://github.com/resque/resque-scheduler

I know it is not quite what you are looking for, but if you looking at that type of cronjob replacement, did you not see a HN link to AirBNB guys talking about their tool Chronos? You might like that.

http://nerds.airbnb.com/introducing-chronos/

If you don't mind running development versions: Notifications can be a part of building that. Problem: you (currently) have to be connected when it happens.

http://redis.io/topics/notifications

Resque Bus uses resque-retry and resque-scheduler to do exponential backoff. That is, it will retry immediately, 2 seconds later, 4 seconds later, and so on up to several hours. It was a little tricky to make sure it ended back in the right application queue, but it worked. If it was put in another one, it would crash because the code to process it would not line up.

Edited a comment from the blog comments a bit...

I tried my best to explain some of the reasons we went this way instead of using things out there. I'm not claiming to have invented anything particularly novel here because it's a simple problem to move things from one list to N lists based on keys. We found value in doing it with technology that was already part of our stack and very well known amongst the team and community.

I did an implementation in RabbitMQ at the outset. I found that the simplicity of just moving the data around, clarity of persistence, and control over subscriptions was worth the relatively short time it took to implement this gem. For example, we have found value in matching on any of the or multiple published keys (specific values, regex, blank, present, not present, etc) as part of the subscription model. I could not figure this out using the routing_key and/or topic kind of stuff in RabbitMQ.

That is close. Thanks for the link.

The Consumer mixin is almost the same as the Subscriber one that I came up with.

Is there a way to subscribe on more than just the topic type? For example, to anything published that has { action: 'received' }

That's something that really enables some great use cases for us.