Hacker News Reader

We tried this at WakaTime and it didn't scale well. We also tried https://github.com/closeio/tasktiger but the only ones that work for us are based on RabbitMQ:

Celery<4.0

https://github.com/Bogdanp/dramatiq

View on HN

Can you please elaborate? I just adopted RQ a few weeks ago and is working fine for me (a few days before launch) with a couple thousand queries (1-2 jobs per 5 seconds) but a ton of connections (both making and taking jobs). Any advice?

I'm thinking that after we go to prod, I'll move to celery and MQ overtime.

This was multiple years ago so I don't remember details. I think something around task reliability, performance, and I think architecture flaws that meant we couldn't prioritize tasks or some other missing features we needed. We process around 50-1000 jobs per second.

Reply on news.ycombinator.com