Why this and not celery?
And why Postgres and not rabbitmq or redis?
And why Postgres and not rabbitmq or redis?
1. For anyone whose stack is currently a simple three-tier architecture that currently has Postgres and nothing else in the DB tier, adding anything else would incur 100+% operational complexification (e.g. now you have to figure out how to do backups for two stateful components.) Far more than 100%, even, if they get the Postgres DB "for free" as part of a virtual LAMP-stack appliance, or built into the base offering of some PaaS service, and currently don't need to do any ops work as the framework/platform handles their DB's care and feeding for them. (Ideally, such setups would do the same with a built-in MQ as well — but sadly, that's much rarer.)
2. You can be clever with a job queue that’s in your DB, by making “taking the job” and “doing the queries that comprise the job” part of the same atomic transaction, such that a ROLLBACK due to a constraint validation error in "the queries that comprise the job" will also implicitly “put back” the job immediately (even if the worker crashed in response to seeing the error.)
If you use Celery, the time saved dropping Celery will more than cover the operational costs of redis.
Really, one of the magic tools of the 21st century.
I chose postrges for our mq stack over rabbitmq and redis. Since our team is small, we are trying to be efficient in our use of manpower. our architecture is as dirt simple as it gets. (loadbalancer -> api-server-nodes -> aws aurora).
adding another external dependency just adds one more thing that can potentially go wrong. Thats one more thing we need to watch and maintain accross staging and production envrionemnts and one more thingto get running on developer machines.
There may come a day where we switch over to rabbitmq or kafka but postgres has proven to be fast enough. when writes are too much of a load, we can shard the writes to its own dedicated machine and buy more time. When our traffic is sufficiently high that even that isn't enough, we'll have enough revenue to pay someone to configure kafka/rabbitmq and deal with the configuration and maintenance fulltime.
Until then, if you want to survive as a startup, KEEP IT SIMPLE. Any complexity you adopt needs to justify itself by providing a competitive advantage.
I love the idea of re-using infrastructure "for free".
I have a production system running on Celery (on Redis). It's been fine, once I tuned it.
Time for a refactor; wondering: should I consider migration?