Hacker News Reader

Comment by CipherThrowaway | Hacker News Reader

How does the idempotency work? I don't understand how functions can be made idempotent without changing or instrumenting their implementation for end-to-end idempotency.

View on HN

You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras.

It would require some client support, though. Alternatively, if the RPCs have the same arguments, the framework could choose to ignore seemingly identical ones, though that has its own issues.

Neither method works.

If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once.

The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.

It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/

> If a machine fails to send any heartbeats within an interval (default 90 seconds):

> It is marked as unhealthy, and Differential will not send any new requests to it.

> The functions in progress are marked as failed, and Differential will retry them on a healthy worker.

I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!

That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.

It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice.

I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...

Is there really consensus that practical systems need to have lock lease timeouts?

It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times.

I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.

Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.

Eventual consistency isn't a philosophy, and it's not the idea that race conditions are acceptable. It's a consistency model with specific properties and guarantees.

These races we're talking about don't produce eventual consistency, but inconsistency. Different systems require different levels of correctness, but if you're not familiar with the underlying theory then your trade-offs are going to be uninformed decisions rather than informed ones.

A race condition is the lack of determinism in a sequence of events that can lead to undesirable effects. Eventual consistency is an approach (and consistency model, design philosophy, whatever you want to call it) that accepts this lack of determinism as a compromise because the possible sequences are accounted for, since the system can still act on stale data until an update is fully pushed out (which is still undesirable, but an accepted compromise).

Ok, I think we could do a better job of explaining the semantics.

The system does not implement idempotency by default. If something that's not marked as "idempotent" fails, it does try again in a different machine.

If the function is marked as idempotent [1] then the system ensure at-most-once semantics by only issuing the call once and only once to a worker who can process the call.

[1] https://docs.differential.dev/advanced/idempotency/

Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/

"Just decorate your idempotent operations" – implies to me that the idempotency of your functions is your responsibility.

That's one way of reading it - although I find it's at odds with the heading "Idempotency in one line of code."

If it's the case that the idempotency wrapper is only for operations that are already idempotent, it makes me wonder that the docs couldn't do a better job of communicating the purpose and function. Decorate your idempotent operations.. to make them more idempotent?

Not quite. The decorator [1] enforces you to provide an idempotency key, if idempotence is needed.

The control-plane intercepts that and implements idempotence for you.

[1] https://docs.differential.dev/advanced/idempotency/

Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.

I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2].

If there's an error in the source code or the copy, I'll happily retract it.

[1] https://github.com/differentialhq/differential/blob/236ffc53...

[2] https://github.com/differentialhq/differential/blob/236ffc53...

Without looking at the code, I can tell you that this will reproduce one of two things:

1. Invoke function that has two side effects

2. Function does first side effect

3. Pull the computer's plug out of the wall

A framework can only do one of two things:

A. Invoke the function again

B. Not invoke the function again

If you choose A, you have at least once processing

If you choose B, you have at most once processing

Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one.

I believe you mention exactly once processing in another comment. It's this. The massive lie that many of these types of frameworks tell is that they can manage idempotence for you. Aside from functions that are side effect free (that is, pure, that is referentially transparent, that is can be made mathematically idempotent) you cannot possibly make them idempotent. It's the two generals problem.

So, I believe what you have implemented is idempotence at the sender with the reservation pattern. This is not the same as idempotence at the processor which is literally the only thing that matters. Idempotence at the sender is simply a performance (or storage, in the case of event sourcing) optimization.

Please correct me if I'm wrong on any of this, but if I am, then clearly my understanding of the two generals problem and distributed systems are wrong and I'm going to have to do some significant rethinking of our system's architecture.

I don't think you're wrong about any of this. I just feel like we're talking over each other. You clearly demonstrate an understanding of the domain, so I'm happy to engage. After all, if we fail, we want to fail fast.

Maybe we can focus the conversation on the scenario you mentioned above.

> Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one.

So, what I'm telling is, yes - the first side-effect (or any side-effect) that you put through the system can be made idempotent through the same tooling that makes the caller idempotent.

  func foo(n1) {
    return networkCall(n1)
  }

  func bar(n2) {
    return networkCall(n2)
  }

  func foobar(n1) {
    n2 = foo(n1)
    n3 = bar(n2)
  }

  func main() {
    foobar(42)
  }

Given the above, the framework allows you to wrap foo, and bar all independently in higher order functions which will not issue a duplicative function calls for the same idempotency key. Therefore, you can call foobar repeatedly, and get the same result.

Understood, and perhaps the difference here is more about what I said in my other comment. I don't see this as particularly useful, and I see it as deceptive. What matters is that when we intend to do something is that it happens. We actually want retries. We want exactly-once processing with at-least-once handling. That's the only way that I know of to build resilient systems.

Our back end components literally crash when they fail. No other processing is done. They crash and if there is a bug they will keep crashing. Because our back end components have autonomy, this results in a very narrow service disruption. We fix the issue, identify the root cause and eliminate it. As a result, we have extremely resilient systems that, even in the face of 3rd party downtimes, we can know that every command that gets submitted will be effected. Dead letter queues or anything of the sort are anathema. We try until we succeed. Idempotence is considered in every single handler all the way from the inside to the outside as it must be, mathematically.

I see elsewhere you mention that you use at most once processing. So, I was possibly mistaken that the function can be called more than once assuming you did this right (cleared the job before even invoking the job). My primary point still stands: this is not idempotent, at least not in the way that matters. This is single-try processing. If all you are doing is returning a synchronous error to the user so that they can retry (and again, every other operation leading up to the one that failed to process is actually idempotent) then this would not be catastrophic. The issue is when people start to think that this actually is idempotence, i.e., they attempt to reuse an idempotence key across requests when the first one straight up failed (or not, who knows).

In other words, don't use the idempotence helper if you care about the thing happening. If it's optional, sure, go for it. You could rename it to "tryOnce", or "maybeRun" rather than "idempotence" and it'd be more accurate / less misleading.

> You could rename it to "tryOnce", or "maybeRun" rather than "idempotence" and it'd be more accurate / less misleading.

This is valuable feedback. Thank you.

Here are the docs on idempotency: https://docs.differential.dev/advanced/idempotency/

You mentioned elsewhere that you're still finding the right wording for the product and its benefits. Fair enough. Personally, I think you should probably stop referring to this as idempotency. Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once. I expect most people with knowledge of - and respect for - distributed systems theory will be completely put-off by this wording.

In distributed systems, the reason idempotency is such a Big Deal™ is that it can be combined with at-least-once delivery to achieve exactly-once semantics. This isn't that. What you're describing as idempotency has the potential to mislead users, especially since you use examples like credit card processing.

Thanks for the thoughtful reply.

You might be on to something here:

> Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once

and we will consider changing it because I agree that conflating the concepts (or the appearance thereof) is not worth diluting the other parts of the offering.

I still struggle to find the difference between this approach and the likes of Stripe [1] and Temporal [2] because in practice, it yields the same result.

[1] https://docs.stripe.com/api/idempotent_requests

[2] https://www.restack.io/docs/temporal-knowledge-temporal-io-i...

One difference is that you are specifically creating distributed systems middleware, where your audience is going to be using your product in the implementation of a system of their own, and is more likely to understand the terminology and wish to know the exact guarantees and trade-offs being provided.

I find both those comparisons are apples to oranges:

1. Temporal does not claim, on that page, to be able to automatically make your calls idempotent. Instead, it is explaining how you - as the user - can and should write your activities to be idempotent. Take the payment code snippet: you can't just wrap `ExternalPaymentAPI.Process(paymentId, amount)` in an idempotency provider. You have to implement specific idempotency logic inside the activity. This is in line with the distributed systems theory of idempotency.

2. In these docs, Stripe describes an interface for achieving idempotency as an external caller. Since Stripe control the implementation of their own system, it is certainly possible that they have implemented their operations in a way that is truly idempotent. What your docs describe - the ability to make arbitrary operations idempotent by wrapping them - is simply not possible.

I wonder whether you're focusing on the idempotency key as the thing that is wrong here. There is nothing wrong with idempotency keys themselves - they are a great way for APIs to provide idempotency. The problem is that this guarantee can't be provided by a wrapper layer, it requires the operation itself to be written in an idempotent way. This is an example of the age old "end-to-end principle" from networks class.

Ok, yes - I'm with you.

I now realise how this statement could be misleading.

> To mark a function as idempotent, simply wrap it with the idempotent function.

When initially writing it, I thought it was a "given" that the wrapped functions are the responsibility of the developer. Now I realise this can also be interpreted as "we make your functions idempotent".

Thanks for the detailed breakdown. We will work on the docs to make it clearer.

If that was the assumption, then this whole thing makes even less sense. If your operation is already idempotent in its implementation, then wrapping it with this function is the last thing you'd want to do. By introducing at-most-once semantics on top of the call, it would defeat the point of making the operation idempotent to begin with.

The scenario where you'd want to use this `tryOnce` policy is probably the following:

1. You have an operation that is not idempotent and can't be made idempotent, for whatever reason.

2. You would prefer the failure mode of that operation to be that the effect doesn't take place, rather than that the effect takes place more than once.

BTW, you don't need to have an answer for everything. I don't think there has been any miscommunication or misunderstanding of the docs. I think people in this thread, including myself, have accurately identified that you haven't fully thought some of this stuff through. That's fine and normal for a product in this stage, but not being willing to admit it is less of a good sign. Good luck with the startup.

> If your operation is already idempotent in its implementation

No, this is not at all what I meant. I don't think there's any value in introducing at-most once semantics to an already idempotent function.

I'm definitely not trying to have an answer for everything. But I guess we can leave it at that, at this point.

Reply on news.ycombinator.com