Comment by sametmax | Hacker News Reader

Comment by sametmax | Hacker News Reader

The TL;DR (tl;dr: this explains how Celery obsoletes GIL and coroutines) is only telling half of the story.

Celery solves some cases for Python:

- background tasks; - tasks queues; - cron tasks.

And it's nice, I used it extensively. But in NO WAY it totally solves the GIL problem nor does it render coroutines obsolete.

Here are some celery limits:

- you need to calibrate the number of worker you have to match your workload.

- You are limited to x number of blocking operations, where x is the number of workers. It means you cannot do massively parallel I/O such as network operations with it (e.g: a web server).

- Tasks don't have access to your main process memory, and vice versa.

- Tasks cannot communicate with each others;

- You must juggle with the workflow of your tasks (is it ready ? it it dead ?). You can use await stuff() with a try/except;

- Celery is an additional process to setup and start, with backends to choose from and tuning to do. It's a lot more work than just importing an async lib.

Granted, celery is a very useful piece of software, but not the silver bullet this article depicts.

> - You are limited to x number of blocking operations, where x >is the number of workers. It means you cannot do massively >parallel I/O such as network operations with it (e.g: a web >server).

Celery can use eventlet/gevent instead of multiprocessing for executing tasks, so this should be possible (granted, not sure if using it as a web server is a great idea)

>- Tasks cannot communicate with each others;

This is not true, they can send messages to each other

>- You must juggle with the workflow of your tasks (is it ready >? it it dead ?). You can use await stuff() with a try/except;

If you have to juggle it means your workflow design is not good enough. I.e. you shouldn't wait for other tasks, you should have callbacks and errbacks.

Also, note that a try/except does not guarantee the operation will be completed (e.g your asyncio app can be killed).

I'm not sure there's much point in comparing these, they are wildly different concepts: Celery is a distributed system, asyncio is for async I/O, the GIL is only a problem if you can't start n instances of your app.

> This is not true, they can send messages to each other

This is a very limited type of communication. Asyncio with stuff like crossbar.io allow pub/sub.

> I.e. you shouldn't wait for other tasks, you should have callbacks and errbacks.

This is the old way of doing something async. Callbacks are hard to reason with, compared to async/await.

> (e.g your asyncio app can be killed)

So can your celery workers. It happened to me many times. What's more, celery setups can cause latency problems which will raise errors like something has died, but no, so very hard to debug.

> I'm not sure there's much point in comparing these, they are wildly different concepts: Celery is a distributed system, asyncio is for async I/O, the GIL is only a problem if you can't start n instances of your app.

Well, the main point of the article is that celery is solving the GIL. It's not, it's bypassing it, providing very important benefits and drawbacks, and can be used nicely for a specific set of tasks. I just want to hilight that.

Asyncio does help to live with the GIL since it unifies threads, coroutines and multiprocessing with the same await/async interface. And asyncio can be used to create distributed system even better than celery setup because of this. Again, crossbar.io comes to mind.

>This is a very limited type of communication. Asyncio with stuff like crossbar.io allow pub/sub.

Celery also supports pub/sub, and other topologies.

>So can your celery workers. It happened to me many times.

With the major difference that your tasks can be redelivered to a different worker, and so will complete anyway.

>Well, the main point of the article is that celery is solving >the GIL. It's not, it's bypassing it

I was agreeing with you there, but I guess my reply was not clear on that. I just wanted to point out some inaccuracies in your reply.

>coroutines and multiprocessing

Be careful using the multiprocessing module, it has some very serious bugs. I've spent the last 4 years rewriting parts of it for Celery

>Be careful using the multiprocessing module, it has some very serious bugs. I've spent the last 4 years rewriting parts of it for Celery

I regretted this as soon as I submitted it. I would hate for someone to do the same thing to my projects so I should know better. I've written about it before, but realize that you probably have not read it :)

I really like the multiprocessing library, it helped me start Celery in the first place. What it tries to solve is actually very very complicated, and you would have to test it on production systems for years to be sure it works, and I think Celery was the app that did that testing. I contributed some fixes back into Python, but most of it is not merged upstream.

The most complicated issue I had to solve was that multiprocessing.Pool uses POSIX semaphores to share pipes between processes (that's how the pool processes receive jobs, and the parent receive results). If a child process is killed before releasing that semaphore you have a deadlock that's tricky, if not impossible to solve. So I rewrote the pool to use async I/O instead, which also had the side effect of drastically improving performance (no locks). Sadly I'm not sure how to implement that on Windows, so it's unlikely to be merged upstream. Other fixes and features used by Celery is available in our billiard (on PyPI) fork of multiprocessing, but the async pool is not part of that yet as it currently depends on code in celery that does not fit in billiard (it should be rewritten to use asyncio now).

You can claim to replace Celery using a small layer on top of async I/O, or claim to replace Celery with a simple Redis list operation, but I think that's unfair to all the work that went into Celery, and the other features Celery implements like monitoring, workflows, and a large list of other things that you don't immediately think of when starting a project. It keeps a repository of these patterns for the Python community, and even something like crossbar.io could be supported as a transport.

I would never claim that you can use "a small layer on top of asyncio to replace celery". I've read celery codebase, it's very, very thorough. Also in the future, I may even try to integrate celery in the asyncio event loop so I don't have to start a separate process.

Hi asksol. I'll take this opportunity for publicly thanking your for all the help you give on IRC on your celery project. You saved my ass a lot of time. You rock.

On Celery limits:

- celery has autoscale which works _awesome_

- autoscale? also, having bounds isn't such a bad idea in reality :)

In regard to what you've written about tasks - yes, and point I was trying to make is that Celery model is a good example of what can be done to enable concurrent programming in Python

Reply on news.ycombinator.com