Hacker News Reader

Comment by mrfusion | Hacker News Reader

I'd like to see something like celery built in. But easier to use and doesn't need a separate db. Is it possinlble to fund that like south was?

View on HN

Django Channels, which is slated for inclusion in one of the next Django releases, covers at least some of the use cases you'd traditionally use Celery for, specifically running code after returning a response. It does require Redis for this though.

https://channels.readthedocs.org/en/latest/

Neat! I couldn't find an example of doing a background task from a view with channels. Any ideas?

Out of curiosity: How would you do that - "easier" and without separate DB?

Not require everything to go into a tasks.py file. And I don't see why such a system can't use the existing db?

If "something like celery built in" means a tool that runs Django as a network service in a concurrent context, with APIs to allow asynchronous processing of tasks, you might want hendrix, which serves Django on Twisted:

https://github.com/hendrix/hendrix

(Others have pointed to django-channels, for which hendrix may eventually evolve into a backend)

Are you looking for something like https://github.com/andrewgodwin/channels

Isn't celery plug and play nowadays? If you need a something simpler take a look at django-rq. I don't think a serious job queue is possbile with RDBMS as backend.

What's so special about a job queue that it can't use an rdbms? I never understood that.

are you asking why RabbitMQ exists?

Yeah, or any documentation that explains the fundamental differences. I get that it's required and I can see the advantages for large scale operations but I never understood why you can run a simpel DB backend for small scale setup.

Just trying to learn here!

You can use relation database but sane implementation is ugly and doesn't scale. There is little reason to choose bunch complexity and unpredictable behavior over dedicated queue dependency. Sorry, i don't have any good material on the topic. Maybe you can use this to see how RDBMS implementation looks and what are the consequences. https://brandur.org/postgres-queues

One reason: Most RDBMS's aren't designed for "waiting for newly queued items". You literally have to have each worker poll at some time interval for new items: `SELECT * FROM jobs WHERE status = 'new'`, transactionally claim the job, and if a worker dies, un-claim it. It's not that it's not possible to use an RDBMS and work around these problems, but generally, it won't lead to the most scalable and robust solution for the problem. Message brokers / AMQP are better designed for this problem set. If you have a small site, with a low volume queue, your RDBMS might be just fine though.

This is not the case if you're using Postgres, which will signal/notify listening clients, instead of having the clients poll for changes.

It wasn't (for me) a year ago when I did an upgrade. I basically had to go through the docs and start from scratch. :(

Reply on news.ycombinator.com