Hacker News Reader

Comment by teagee | Hacker News Reader

Any code or blog written by Adam is worth spending some time on.

It will be interesting to see how the tasks framework develops and expands. I am sad to see the great Django-Q2 lumped in with the awful Celery though.

View on HN

Celery is the worst background task framework, except for all the others.

There are bugs and issues, but because so many people are using it, you’re rarely the first to stumble upon a problem. We processed double-digit millions of messages daily with Celery + RabbitMQ without major obstacles. Regardless of what people say, it should be your first go-to.

I think Celery has a lot of magic happening under it. When the abstractions are so high, it's important they never leak and you don't see anything below the turtles you are supposed to see.

I often prefer designing around explicit queues and building workers/dispatchers. One queuing system I miss is the old Google App Engine one - you set up the queue, the URL it calls with the payload (in your own app), the rate it should use, and that's it.

Celery has way too much magic crammed into it, it is very annoying to debug, and produces interesting bugs. Celery is/was also a "pickle-first" API and this almost always turns out to be the wrong choice. As a rule of thumb, persisting pickles is a really bad idea. Trying to hide IPC / make-believe that it's not there tends to be a bad idea. Trying to hide interfaces between components tends to be a bad idea. Celery combines all of these bad ideas into one blob. The last time I looked the code was also a huge mess, even for old-guard-pythonic-code standards.

OP here, thanks for the praise!

Yeah, I mentioned Celery due to its popularity, no other reason ;)

You are a great writer - thanks for putting this together!

I tried django-q and I thought it was pretty terrible. The worst was that I couldn't get it to stop retrying stuff that was broken. Sometimes you ship code that does something unexpected, and being able to stop something fast is critical imo.

Fundamentally I think the entire idea behind celery and django-q is mostly misguided. People normally actually need a good scheduler and a bring-your-own queue in tables that you poll. I wrote Urd to cover my use cases and it's been rock solid.

I've been using Celery for years. What is the major issues you have with it and how does Django Q2 help?

I also use Kafka on other tech stacks but that's another level completely and use case.

Why is celery awful?

Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times.

And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.

https://flower.readthedocs.io/en/latest/

Although we could say the same thing about Kafka, couldn't we? It's made for much higher throughput and has usually other use cases, but it's also great until it's not great.

At least the last time I used Kafka (which was several years ago so things might have changed) it wasn't at all easy to get started. It was a downright asshole in fact. If you pursue a relationship with an asshole, you shouldn't be surprised when they become cold to you

Yes, absolutely. It's still pretty much that way. Especially if you want to make changes to a running installation, add nodes etc.

Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:

- your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes

There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.

Honestly, it's a great education.

From your experience, what is a better alternative guys?

There’s no alternative (while prototyping), and anything else is better (when you properly defined your case).

DjangoQ2 is a fine alternative during early development

Not the comment that you replied to but I use my own Urd. It's a fancier Cron that you can stop fast. Which is imo what you normally want.

Task queues are like email. It's what everyone is used to so people ask for more of it, but it's not actually good/the right tool.

> your side effects (e.g. database writes) aren't idempotent

What does idempotent mean in this context, or did you mean atomic/rollback on error?

I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips

In my experience async job idempotency is implemented as upserts. Insert all job outputs on the first run. Do (mostly) nothing on subsequent runs. Maybe increment a counter or timestamp.

In the context of background jobs idempotent means that if your job gets run for a second time (and it will get run for a second time at some point, they all do at-least-once delivery) there aren't any unfortunate side effects to that. Often that's just a case of checking if the relevant database updates have already been done, maybe not firing a push notification in cases of a repeated job.

If you need idempotent db writes, then use something like Temporal. You can't really blame Celery for not having that because that is not what Celery aims to be.

Temporal requires a lot more setup than setting up a Redis instance though. That's the only problem with it. And I find the Python API a bit more difficult to grasp. But otherwise a solid piece of technology.

With Temporal, your activity logic still needs to ensure idempotency e.g. by checking if an event id / idempotency key exists in a table. It's still at-least-once delivery. Temporal does make it easy to mint an idempotency key by concatenating workflow run id and activity id, if you don't have a one provided client-side.

Here is a nice guide from AWS https://docs.aws.amazon.com/wellarchitected/latest/framework...

It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).

> The Many Problems with Celery:

— https://steve.dignam.xyz/2023/05/20/many-problems-with-celer...

> The problems with (Python’s) Celery:

— https://docs.hatchet.run/blog/problems-with-celery

> Dramatiq motivation:

— https://dramatiq.io/motivation.html

Here are some alternatives:

Dramatiq: https://github.com/Bogdanp/dramatiq

RQ: https://github.com/rq/rq

Huey: https://github.com/coleifer/huey

Hatchet: https://github.com/hatchet-dev/hatchet

django-q2: https://github.com/django-q2/django-q2

Would you consider tools like Temporal, DBOS, Absurd Workflows, PGQueuer as alternatives?

https://temporal.io/

https://docs.dbos.dev/

https://news.ycombinator.com/item?id=45797228

https://python-absurd-client.readthedocs.io/en/latest/quicks...

https://pgqueuer.readthedocs.io/en/latest/

Temporal is an AMAZING piece of software, however I don't believe it's a replacement for something more simple like Celery. Even if you write helpers, the overhead to setting up workflows, invoking them, etc. is just too much for simple jobs like sending an email (imo). I would love to work in a codebase that had access to both, depending on the complexity of what you're trying to background.

I’m currently stuck with the tech debt of Celery myself. I understand that! Does Django Tasks support async functions?

Computer, load up Celery Man please.

I'm of the opinion that django task apps should only support a single backend. For example, django-rq for redis only. There's too many differences in backends to make a good app that can handle multiple. That said, I've only used celery in production before, and I'm willing to change my mind.

With that logic, the Django orm should only support one database.

Why have a backend then?

Reply on news.ycombinator.com