Hacker News Reader

Anyone want to chime in on why Python threads and subprocesses can't be used?

Subprocesses aren't used because the cost of forking is high. It's more efficient to fork a worker once, then have that worker keep doing work.

The problem now becomes transferring data to that worker, which is where celery comes into play.

Python threads cannot be used for CPU heavy loads because they all run on a single core.[0][1]

[0]: CPython threads are kernel threads but effectively act like user threads. All threads run within a single VM and thus restricted to a single core.

[1]: Jython does not have the same limitation.

IIRC, unless you explicitly state otherwise, Celery forks for every task. A lot of task queues do this because it becomes a lot easier to recovery from errors.

Multiprocessing makes it pretty straight-forward to manage inter-process communication with queues, but you'd have to write all the other stuff task queues handle for you, such as error management.

Reply on news.ycombinator.com