Hacker News Reader

Comment by simonw | Hacker News Reader

Well I love Celery, so I'm even more excited to learn more about Faust now :)

I'm looking at Redis Streams and it seems to lack the partitioning component of Kafka. One of the properties that we rely on is the ability to manually assign partitions such that a worker assigned partition 0 of one topic, will also be assigned partition 0 of other topics that it consumes from.

Simplicity is of course a goal, but this may mean we have to sacrifice some features when Redis Streams is used as a backend.

I'm glad you like Celery, this project in many ways realize what I wanted it to be.

If you liked Celery 3.x but don't like all the bugs in Celery 4.x, check out these alternative Python task queues:

https://github.com/closeio/tasktiger (supports subqueues to prevent users bottlenecking each other)

https://github.com/Bogdanp/dramatiq (high message throughput)

In Faust we've had the ability to take a very different approach towards quality control. We have a cloud integration testing setup, that actually runs a series of Faust apps in production. We do chaos testing: randomly terminating workers, randomly blocking network to Kafka, and much more. All the while monitoring the health of the apps, and the consistency of the results that they produce.

If you have a Faust app that depends on a particular feature we strongly suggest you submit it as an integration test for us to run.

Hopefully some day Celery will be able to take the same approach, but running cloud servers cost money that the project does not have.

Reply on news.ycombinator.com