Hacker News Reader

Comment by memset | Hacker News Reader

Here is a question: I often find myself wanting to do background tasks within my application (say, a Flask app) without running celeryd. While celery is quite useful, I just want to spin up an application and have it Just Work without external dependencies. With celery, now I have to run celeryd, run some sort of queue (redis, rabbitmq, etc), and just worry about all of these pieces simply to execute long-running background tasks.

Are there options for this? Akin to an sqlite for tasks, in python?

View on HN

Running your Flask application with uWSGI could allow you to do just that by way of the spool and other related decorators: http://uwsgi-docs.readthedocs.org/en/latest/PythonDecorators...

+1 for uWSGI. The Spooler is awesome and also, if you need lower-level stuff, have a look at its SharedArea and Queue[1] frameworks, as well as Mules[2].

[1] http://uwsgi-docs.readthedocs.org/en/latest/Queue.html [2] http://uwsgi-docs.readthedocs.org/en/latest/Mules.html

I've read about https://github.com/tthieman/dagobah recently, but I haven't used it...also, if you think celery is too heavy, you'll probably think this is too. I've also heard good things about https://github.com/binarydud/pyres.

Despite all that, I usually just find myself running celery. I'll usually have a redis instance going anyways that I can use as a sub-optimal queue, and if it's a small enough site, I'll run a single worker instance on the same machine as the host.

Advanced Python Scheduler (APScheduler) is a light but powerful in-process task scheduler that lets you schedule functions (or any other python callables) to be executed at times of your choosing.

This can be a far better alternative to externally run cron scripts for long-running applications (e.g. web applications), as it is platform neutral and can directly access your application’s variables and functions.

I've been using Python RQ (http://python-rq.org/) as a lightweight alternative to celery. It uses Redis as its backend so I guess that might not meet the "SQLite" qualification, but has been light weight enough for my application (e.g. no reqt for a message queue like rabbit). Really liking it so far.

We're using RQ in production and are very happy with it. We chose it over Celery because the "protocol" is much more straight-forward than Celery - when something fails, I can peek at the state of redis or the RQ source code and know what's going on.

The only downside is that, like Celery, RQ forks for every task. This can feel a bit heavy for some of our simpler tasks, although it hasn't proven to be an issue yet.

>While celery is quite useful, I just want to spin up an application and have it Just Work without external dependencies. With celery, now I have to run celeryd, run some sort of queue (redis, rabbitmq, etc), and just worry about all of these pieces simply to execute long-running background tasks.

You can use redis as a backend for celery (which you should be using already for caching, probably), and AFAIK you can use a database as a backend too (although I haven't tried it).

That means that celery is exactly one dependency.

You need some kind of persistent store so that it can recover the task queue from hardware failure. But it can probably use the backend you already have.

APScheduler is a nice in-process alternative that doesn't require running another process or using a db for persistence. Quick to start using and no external dependencies. https://pypi.python.org/pypi/APScheduler/

Celery can use a database backend, such as Postgres. Maybe it can work with sqlite?

You definitely can use celery with sqlite via the sqlalchemy backend, but since rdbms' aren't very good task queues there are drawbacks: see http://docs.celeryproject.org/en/latest/getting-started/brok...

That's true. However, as an answer to the above it may well be acceptable without needing a different library.

Anyone want to chime in on why Python threads and subprocesses can't be used?

Subprocesses aren't used because the cost of forking is high. It's more efficient to fork a worker once, then have that worker keep doing work.

The problem now becomes transferring data to that worker, which is where celery comes into play.

Python threads cannot be used for CPU heavy loads because they all run on a single core.[0][1]

[0]: CPython threads are kernel threads but effectively act like user threads. All threads run within a single VM and thus restricted to a single core.

[1]: Jython does not have the same limitation.

IIRC, unless you explicitly state otherwise, Celery forks for every task. A lot of task queues do this because it becomes a lot easier to recovery from errors.

Multiprocessing makes it pretty straight-forward to manage inter-process communication with queues, but you'd have to write all the other stuff task queues handle for you, such as error management.

Reply on news.ycombinator.com