Celery was never reliable for me. I used Celery as a part of Django. It kept running into weird issues. Celery-Flower used to monitor the Celery threads is not maintained and that became a huge issue for me as I had no way of tracking any issue.

I have also used Celery parallelize crawling of multiple websites in parallel. Beyond a small number of workers, Celery would go into a frozen state (My data flow was one way. No chance that I know of deadlocking.) The code felt too convoluted to dive in and understand.

I have since shifted to Dramatiq ( https://dramatiq.io/ ). It has all the features I like of Celery (Chaining of async methods with error handling when any one link on the chain fails), with super reliability. It also has Django support.

Having said that, it must be recognized that Dramatiq was built from the success and learnings gained from Celery. Celery is a wonderful piece of work still in active use in almost all Python shops. I am just thankful that we now have so many alternatives.

PS: The Django-Q mentioned in another comment looks very well structured, comprehensive and actively developed. I shall give it a try. For now, so happy with Dramatiq.