Comment by _pastel | Hacker News Reader

Comment by _pastel | Hacker News Reader

Word of warning - if you write your own task queue at a startup, you will spend the rest of your tenure justifying this decision to every new data engineer who joins.

Also, am I crazy or do the celery docs not even clarify their delivery semantics? Isn't that table stakes for a queueing system? As best as I could tell you can get close to "at least once" with

  acks_late=True, task_reject_on_worker_lost=True

but not in cases like a worker hanging indefinitely without being explicitly killed.

My opinion on this is that if you need at least once semantics it is best to encode this behaviour in a store you can trust, take the ID of the task and then schedule a task with that data.

Then, add a "sweeper" that checks if the task actually happened and if it did not happen, requeue the job.

To ensure proper behaviour that the same job is not being worked on twice, add some locking in there and you have a really stable way of scheduling jobs.

Could this have been done INSIDE celery? Sure, but I would say that this is not the case that it should optimize for.

Every version of this I experienced went like this:

> We have a task queue…

> Is it Celery?

> No

> oh thank god. How do I use it?

> Put the @task decorator on a function and call it. See the args to that decorator if you want slightly more control.

Reply on news.ycombinator.com