Hacker News Reader

Comment by scarmig | Hacker News Reader

You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras.

It would require some client support, though. Alternatively, if the RPCs have the same arguments, the framework could choose to ignore seemingly identical ones, though that has its own issues.

View on HN

Neither method works.

If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once.

The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.

It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/

> If a machine fails to send any heartbeats within an interval (default 90 seconds):

> It is marked as unhealthy, and Differential will not send any new requests to it.

> The functions in progress are marked as failed, and Differential will retry them on a healthy worker.

I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!

That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.

It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice.

I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Is there really consensus that practical systems need to have lock lease timeouts?

It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times.

I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.

Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.

Eventual consistency isn't a philosophy, and it's not the idea that race conditions are acceptable. It's a consistency model with specific properties and guarantees.

These races we're talking about don't produce eventual consistency, but inconsistency. Different systems require different levels of correctness, but if you're not familiar with the underlying theory then your trade-offs are going to be uninformed decisions rather than informed ones.

A race condition is the lack of determinism in a sequence of events that can lead to undesirable effects. Eventual consistency is an approach (and consistency model, design philosophy, whatever you want to call it) that accepts this lack of determinism as a compromise because the possible sequences are accounted for, since the system can still act on stale data until an update is fully pushed out (which is still undesirable, but an accepted compromise).

Ok, I think we could do a better job of explaining the semantics.

The system does not implement idempotency by default. If something that's not marked as "idempotent" fails, it does try again in a different machine.

If the function is marked as idempotent [1] then the system ensure at-most-once semantics by only issuing the call once and only once to a worker who can process the call.

[1] https://docs.differential.dev/advanced/idempotency/

Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/

Reply on news.ycombinator.com