Hacker News Reader

Comment by tonyhb | Hacker News Reader

Hot take: I don't think that people should even use queues any more. It's like using raw HTTP2, or managing threads.

Use a workflow engine which abstracts queues for you so you can "just write code" without managing jobs/schedules/state/retries.

https://www.inngest.com/blog/how-durable-workflow-engines-wo...

(disclaimer: I'm biased as I'm an author of a workflow engine)

View on HN

In the example in your post, the queue is abstracted away in some sense, but you aren't just "writing code". There's step.run(), step.sleep(), step.sleepUntil() sprinkled throughout. I'd say in something like Celery, which is explicitly about running jobs asynchronously backed by a message broker/queue, you really do just write code, but then call the function with .s().apply_async() and so on. Now, I'm not saying you're wrong that we ought to abstract queues away, just pointing out that the abstraction leaks in your workflow example. Happy to hear where/how I'm wrong on this.

It leaks in that the entire function becomes declarative, and state is colocated to a single function. You don't have multiple jobs for each attempt, passing indexes into each job queue. You don't have to worry about enqueueing each "step" as a separate job.

The code becomes "wait until this time", vs "enqueue this function with this state to run at this time via this message broker, which may enqueue other jobs in ways you can't see".

There's no real silver bullet that will let you run code in the future without specifying "when" that code should run — but workflow engines are by far the more productive of the two.

Check out temporal.io that fully abstract this. Disclaimer, I'm one of the founders.

hah, I was thinking about temporal as I was writing this. I have played with temporal pretty extensively.

Reply on news.ycombinator.com