Large Problems in Django, Mostly Solved: Delayed Execution | Hacker News Reader

Large Problems in Django, Mostly Solved: Delayed Execution

ericholscher.com

58 pointsmnemonik14 comments

Celery together with RabbitMQ (or another message broker) is great for big/huge projects. For basic job queueing and delayed job execution, beanstalkd (http://kr.github.com/beanstalkd/) is in most cases a better choice. It takes almost no memory, no configuration, and just works.

I wrote a small library for Django that makes writing jobs for beanstalkd easier (http://github.com/jonasvp/django-beanstalkd). There's also a management command for starting (threaded) workers. Makes writing and calling jobs from Django very simple.

Could you please elaborate why it is a better choice? I don't see how django-beanstalkd could take much less memory than celeryd. If it's about the broker (beanstalk), you could have implemented a beanstalk backend for celery as well, then you would have had a tested implementation of the worker part.

Btw, If you would be interested in writing a beanstalkd backend for celery I would be happy to help you out!

Good point - I was under the impression that celery _only_ works with an AMQP message broker. Guess I should have read the description more closely before starting on django-beanstalkd.

I'm happy with my lightweight model for now but I'll definitely look into celery again, when the need arises.

I can personally attest to the fact Celery is pretty awesome, and a nice compliment to your Django app. The author - Ask is also doing some awesome things with multiprocessing in general - check out http://pypi.python.org/pypi/billiard/

I use Celery myself as well and it's awesome.

It supports multiple messaging queues and backends for storing results (RabbitMQ, MongoDB, AMQP, Redis, ... - I personally use Redis, because I don't need all the features and overhead from RabbitMQ) and also has a great documentation and friendly support on IRC.

And if you watch the Celery project on github you can see that the author is basically working and improving the project on daily basis.

We've been using Celery with Django for a solid 8 months or so. It's been a blessing for our needs, and extremely easy to scale.

How does celery differ in functionality than a message queue like RabbitMQ?

Celery leverages a message queue (including Rabbit, which is how we deploy it), Celery provides an easy API to define tasks, push them onto your queue, and run them on workers.

Do scheduling tools in a web framework protect you from having to write threading code yourself? Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code?

Also, is that like Quarts in the Spring Framework?

> Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code?

This isn't about threading, really. I'll explain it in terms of Rails's delayed_job because I'm not familiar with the details of Celery:

    * Let's say I want to send an email, via the function Mailer.deliver_notice(email)
    * With delayed_job, I turn this call into Mailer.delay.deliver_notice(email). This puts an entry into a Jobs table in the database that makes note of what I want to run, with all the details.
    * In the background, I have a daemon running that checks the Jobs table every five seconds for jobs that need to be run. It uses the information in the table to execute the deliver_notice function.
    * That's it.

In many cases, the job will update some sort of field in your model that you can query to see if it's been run yet or not.

I'm a big fan of doing this with Gearman (http://gearman.org/). Super simple interface, very easy to shove stuff into the background.

It's a little off topic but I would like to take a moment and profess how much I LOVE the fact that hacker news has so many articles on all languages, popular or not.

What are the advantages/disadvantages of using a messaging framework like Celery vs. handling asynchronous tasks in threads?

Celery workers can run on any networked computer - it can be scaled infinitely!

Reply on news.ycombinator.com