Hacker News Reader

Comment by emmanueloga_ | Hacker News Reader

I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think?

A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stitched together on top of queues, but a DE workflow gives you the same guarantees with far less complexity.

The main tradeoff I can think of is latency: DE engines add overhead, so for very high throughput, huge fan‑out, or ultra‑low‑latency pipelines, a bare‑bones queue + custom consumers might still be better.

Curious where others draw the line between the two.

View on HN

Drawing the boundary at high throughput, huge fan-out and ultra-low-latency is correct - I'd also add that MQs are often used for pub/sub and signaling.

MQs are heavily optimized for reducing E2E latency between publishers and consumers in a way that DE engines are not, since DE engines usually rely on an ACID compliant database. Under load I've seen an order of magnitude difference in enqueue times (low single-digit milliseconds for the MQ p95 vs 10ms p95 for Postgres commit times). And AMQP has a number of routing features built-in (i.e. different exchange types) that you won't see in DE engines.

Another way to think about it is that message queues usually provide an optional message durability layer alongside signaling and pub/sub. So if you need a very simple queue with retries _and_ you need pub/sub, I'd be eyeing an MQ (or a DE execution engine that supports basic pub/sub, like Hatchet).

I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution

( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet )

It doesn't make sense to persist by default. If I send a message to my rendering thread's message queue that the window is fully occluded I never want that message to be persisted. If the process crashes the fact that the window was occluded back then has no relevance to the current insurance of the process.

Trying to persist things has a performance cost that you don't want to pay everytime you want a thread to communicate with another.

Do you think ArrayList or std::vector should be persisted by default?

Highly biased opinion here since I'm the CEO of DBOS:

It'll be rare that the overhead actually has an effect, especially if you use a library like DBOS, which only adds a database write. You still have to write to and read from your queue, which is about as expensive as a database write/read.

Reply on news.ycombinator.com