Hacker News Reader

Comment by andybak | Hacker News Reader

Side niggle - I used to notice a lot of Django projects would use complex job queues for absurdly low workloads. Beginners would get recommendations to use RabbitMQ and Redis for sites that probably were only going to see a few hundred concurrent users at most.

Seriously - don't add complex dependencies to your stack unless you need them. The database makes a great task queue and the filesystem makes a great cache. You really might not need anything more.

View on HN

The problem with that is that, even though PostgreSQL has some great functionality for pub/sub and queues, most Django queue systems don't use them by default. Most of the ones I've seen resort to polling (which might be fine, depending on the use case), which is suboptimal.

Otherwise, I agree, I would love to be able to use Postgres for queues, and I think there is a Dramatiq Postgres backend that will do queueing properly (EDIT: Yes, Dramatiq-PG).

Postgres PubSub is great for triggering job dispatch, but it isn’t good enough on its own. There are a few caveats that make polling an additional requirement:

1. Message compaction — notifications are deduplicated, so if two jobs are inserted in the same queue the system may only get one notification. 2. Message saturation — with high levels of activity you’ll need to start discarding messages, essentially denouncing, otherwise the database gets throttled. 3. Dedicated connections — pubsub requires a dedicated connection for listening, which requires a single connection with custom dispatching or one connection per queue.

Relying on PG for everything is awesome regardless!

Is this a problem for low-traffic scenarios? My thinking is basically that I should save the extra dependency until I have more than a few tasks per second, at which point I will switch to something like RabbitMQ.

I don’t think it is much of a problem until you are pushing 2-3k jobs a second. Only the message saturation is a bottleneck anyhow, everything else I mentioned is handled by how a library is architected.

That makes sense, thanks. I generally switch long before I get anywhere near to where the database is the bottleneck, I use Postgres as a broker only for low-traffic sideprojects, where it's much more worthwhile to not bother with an extra broker.

Yeah, that was my issue. Seemed like a ridiculous amount of stuff just to process a few jobs once a day in the background. Eventually I found django-background-tasks and have bene using that ever since. It uses whatever database you're already using, so very quick and easy.

At a glance, https://github.com/lilspikey/django-background-task indeed seems like a good and easy way to get started. You can later switch to something more specialized once you hit the right scale or when you need more sophisticated features.

Just FYI, this fork is updated https://github.com/arteria/django-background-tasks

Yep. That's the solution I've been recommending over Celery for most people.

Half the time people seem to reach for Celery etc all they want to do is run a long task without blocking the response. That's a helleva lot of code to add to your project to just have that functionality.

When I wanted to add processing for background jobs to a Django project, the most obvious solution was Celery, and the most obvious job queue for Celery was Redis.

I looked for a solution that would use Postgres+Celery but there wasn't anything obvious. There is a way to just use the filesystem for Celery, but that doesn't seem ideal.

So that's how I ended up with a Redis dependency.

Exactly my complaint - it's become the "obvious" way to do it than involved several 10s of thousands of lines of code (Celery+RabbitMQ+Redis) plus binary dependencies and associated background daemons.

You can roll your own background task in a few dozen lines of pure Python.

for a long time there were no easy way to do background processing in django. with the asyncio paradigm spreading throughout the community and django starting to add support for it, that will quickly change.

> for a long time there were no easy way to do background processing in django.

There was no easy way to do complex background tasks - but Django is just Python. Spawning a new process is easy as is scheduling via cron or similar. I've used both to implement simple background tasks.

Reply on news.ycombinator.com