What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.
You need to store the payment state at each relevant step and process it asynchronously. If requests time out, you check the status of it using the key you store (with the processor) to see if it was even received.
It’s not perfect, some processors will 500 while processing the payment (Braintree), so you still need reconciliation on the backend.
Regardless, I think your assumption about how the request/response cycle should be working is wrong. For this kind of API and transaction, the server should be returning a response immediately: 202 Accepted. The only thing the API server should be doing before returning is creating a row in a DB (with a "state" field with an initial value of "pending"), and pushing some work on a queue.
The server should not be sitting there with the HTTP request open, trying to complete the transaction, and only returning a response to the client when the transaction is finished or has encountered an error.
The client will have to learn about the progress of the state of the transaction outside of this initial request. There are many options here: polling, webhooks, a message queue like kinesis or kafka, etc.
Idempotency-Key should not replay the response (it depends, actually). But also it should not error 409. You need to be content aware before adding Idemmpotency Key header handling.
What will happen when the request is received and handled but during writing response body TCP connection dropped unexpectedly. And after second or two a connection reestablished. How two sides agree that previous request accepted and everything good to go? That's what Idempotency-Key header does.
An HTTP request comes in with a certain idempotency key. The server returns 202, as you say, and begins to process the database transaction.
While the server is still procesing the database transaction, a second HTTP request comes in with the same idempotency key. What response does this second HTTP request get? The original transaction that the first HTTP request triggered hasn't succeeded and hasn't failed, so it doesn't fall into either of the categories in the post I responded to.
Your answer is that the second HTTP request gets a 409, which makes sense to me, although others are objecting to it.
You seem very focused on long-running orchestration type systems. You build these on top of basic transactional primitives, but it's a mistake to try to make the whole process a single transaction. You can have a quick, transactional "start process" operation which must be idempotent. Other operations like "check status" need not be so complicated.
You don't necessarily share the idempotency key between the "start process" request and the "check status" request. You could for convenience, but it isn't necessary, and on balance most APIs don't. This is the "client picks ID" vs "server picks ID" design choice.
Fair enough. So basically your approach is to wait until the first request completes to decide how to respond to the second request that came in with the same idempotency key.
However, that would seem to me to imply that when the second request comes in, you check its idempotency key, realize you've already received a request with that key and you're processing it, and don't do anything else with the second request until the first one is completed. In particular, you don't have the second request trigger the start of another transaction.
But elsewhere in this thread, you've said you would start a second transaction based on the second request, and let your database's transaction mechanism tell you that it's a duplicate when you try to commit it. Why would you do that if you've checked the second request's idempotency key and you know it's a duplicate?
> You seem very focused on long-running orchestration type systems.
I'm not focused on anything except getting what I thought would be a simple answer to a simple question. The above seems to provide that (though it still leaves a question open, as above). That's all I wanted.
> You don't necessarily share the idempotency key between the "start process" request and the "check status" request.
I'm not talking about a "check status" request. The scenario I've been asking about all along is when a second "start process" request comes in with the same idempotency key as a previous "start process" request, while the process is still in progress.
Part of the problem here is that we're confusing how do you structure the API (replay? 409? something else?) with how we implement the API. The original article (and my original response) focused on API structure. We're wandering into the details of implementation, which is fine, but there are of course many ways to do the implementation. Some simpler than others.
Here's the simplest and most reliable way to implement idempotency for a trivial "create payment" operation, where the client submits an idempotency key. This pattern is incredibly common. Every request looks something like this:
* Start a transaction
* Lookup "does this idempotency key already exist"
* If it doesn't, insert the payment record with the idempotency key
* Commit the transaction
* Return the result. Successful insert is always 200OK. "key already exists" results in either replay of the original result (Stripe model) or an explicit error like 409 (my favored approach, still ubiquitous in ecommerce, and very common in financial APIs that predate Stripe).
Does that help? If you're using your database to handle concurrency, you need every request to start inside the transaction. You can't check the idempotency key outside of the transaction or you can't guarantee once-and-only-once behavior.
[Before someone mentions it, yes you can use a unique constraint instead of an explicit transaction, and this is conceptually identical - the check-for-dup transaction is inside a single INSERT]
What you said up to that point didn't really. But then you said this:
> If you're using your database to handle concurrency, you need every request to start inside the transaction. You can't check the idempotency key outside of the transaction or you can't guarantee once-and-only-once behavior.
Which answers the question that what you said earlier in your post raised. If I'm understanding you right, "lookup the idempotency key" is also relying on the same database, so you need the whole operation to be inside a single transaction in that database.
It would seem to me that you would want "what happens if a second request comes in with the same idempotency key while the first is still in progress" to be part of the API, so clients would know what your server is going to do in that scenario.
You could invent your own more sophisticated idempotency API but good luck finding someone that wants to implement it or use it. What real-world problem are you trying to solve?
Meaning, clients don't care about the thing I asked about?
> What real-world problem are you trying to solve?
I'm trying to understand your answers to my questions. When there seems to me to be something missing, I ask about it.
No no no no no.
You have multiple clients submitting the same business operation simultaneously. One must succeed, the others must fail. If you're using the 409 approach ("notify client that request is redundant") you must not send a 409 code until the work is complete.
The client must interpret 200 and 409 as success cases. 200 means "it was done" and 409 means "it was already done". Clients looping (say, processing durable queue messages) can stop when they receive these responses.
If the work is not complete, you can't return 409, or clients will think the work is done. You will lose messages.
No, it shouldn't. The comment you're responding to is taking 200 to mean "success" and 409 to mean "it was done" so if it was not in fact done then you _must not_ return that.
That said, I thought one of the benefits of idempotency was nonblocking APIs so I'm not sure I like that scheme. It seems like 200 should mean "submitted, accepted, incomplete" and 409 should mean "previously completed". The client never knows which request succeeded but they're idempotent so that doesn't matter. You just poll until the 200 becomes a 409.
Of course that would provide zero diagnostics in the case of failure so I think it's not sufficient as described.
It doesn't have to be multiple clients. It could be the same client, not having received a response to its first request and deciding to re-send the request again.
But, rather than 409, I'd say that you should be using opportunistic concurrency control if you adopt this perspective. There should be a resource context for the request, so the client can obtain an ETag and send If-None-Match headers, and get a 412 response if things are out of sync. That allows them to retry a failed/lost request and safely prevent a double action.
Under a 412, they have to step back and retry a larger loop where they GET some new state and prepare a new action. Just like in DB transaction programming, where your failed commit means you roll back, clean the slate, and start a whole new interrogation of transaction-protected state leading up to your new mutation request.
That doesn't mean that idempotency keys have to be used. You can certainly hash message content if that is documented behavior. That probably only makes sense when there is already some logical session or transaction identifier that makes dedupe semantics clear.
The system you propose might be sound and might be necessary in some systems, but I can't think of what they might be that wouldn't be better served by the simpler solution that is already widely used for this purpose.
If it processed 99% of the request and the final bookkeeping failed because of a duplicate, that's still a failed request.
Arguably this should be the primary way you check for idempotent requests - you shouldn't have a separate check for existence, you should have the insert/update fail atomically.
This is the same thing you see on filesystems for TOCTOU security holes - the right way is to atomically access and modify once, and you only know the request was already processed because that fails.
The fact that payments have a settlement process is not relevant to this discussion.
Yes, I agree. You want to generate a token, persist it locally and use that to communicate with the payment gateway, so re-submissions use the same key and either error or return the transaction state.
> The fact that payments have a settlement process is not relevant to this discussion.
I wasn't talking about settlement, I was talking about the processing aspect. What I meant was: once you kickstart the process with the gateway, money is highly likely to change hands as a result. This means a process of:
1. POST /checkout
2. Create token
3. POST to payment gateway with token
4. Wait for gateway to return
5. Persist transaction/error
6. Return success/error
What is needed is to persist and return the token to the caller before contacting the payment gateway, to make a check + retry mechanism possible.
And yes, I've seen code that follows steps 1-6 exactly as I've described and, yes, all the problems you imagine would occur from those steps have occurred at one time or another.
Even then, it’s not fine because those requests might time out, or your request times out waiting for theirs. Just because your provider abstracts behind one API doesn’t mean you necessarily can!
500 errors, network timeouts, etc all happen. We can't run 2PC transactions with Stripe, so you need durable retries. People run billions of dollars through these APIs every day. It's fine.
My point is that you can’t rely on this specific mechanism because request failure does not mean the payment is not going through!
I’m not sure where we disagree so I must ask: do you disagree with what I’ve written and, if so, what and why?
Even if you have a complex long-running multistep orchestration problem, you can break it down into simpler transactions. Eg you could start with a "lock the resources" txn.
But 99% of these conversations around idempotence are simple POST operations like "create order" that regular old database concurrency management handles just fine.
That doesn't answer my question. What response do you return to the client in the case I described?
So one will complete with 200, one will complete with 409. It doesn't matter which.
That said, there's something odd about the way you phrased this question. If the original request hasn't gotten a response yet, why is it sending a retry? What you're asking is more general: What happens when two conflicting requests come in? This is something we've been solving with RDBMSes since the 1970s.
Because it hasn't gotten a response yet. That's got to be far and away the most common reason any request gets retried in any context.
> why is it sending a retry?
may be two clients tries to do it? Or there's a bug with the client in how they do it?
Isn't the point of idempotency meant to enable clients to retry again, without fear that a 2nd request somehow breaking things?
You absolutely must wait for one request to finish before any other request can return a 409. 409 is a signal to the client that they can stop retrying, the job is done. If some request returns 409 early and the "original" request fails, you will not get further retries and the message will be lost.
Stripe's approach requires serialization as well. Only one request can succeed. If you send multiple conflicting requests in simultaneously, some of those have to block.
The good news is that we have been solving this problem for decades and we have incredibly well refined tools - database transactions and isolation levels - for solving this problem.
And you haven't considered multiple servers in your scenario - what if two requests meant to be idempotent with each other arrived at different servers?
And at the sake of repeating the above commenter, you solve the multiple server by serializing somewhere, because you ultimately need a lock on something. You can also perform the operation in both places and then reconcile the state later but that’s a lot more complex.
When you are using TCP, and you send the same data twice because of a delayed ack, you likewise don't care if the ACK is for the first time or the second time you sent the data. You just know the other side got the data, and that's all you care about.
By sending a third request and getting a response that reveals the state of the system.
Not necessarily - there are different transaction isolation and conflict resolution methods provided by every database built for this purpose. You just have to ensure that only one request actually commits to the database, and that one sends a success response while the other sends a 409. The database or another lock provider can either help enforce serialization up-front - or the app can use optimistic locks based on data in the request that will only block if there is actually a conflict, and this won't delay the first transaction at all.
Solving these kinds of issues are exactly the purposes of idempotency keys and database transactions and using them in the intended way is really the only sound way to build a distributed system. Making things more complicated to "improve DevX" is just going to make them unsound. That is what Stripe chose to do. Their 24-hour replay idea is fine but why not send 409s after that rather than accept those transactions? If "that will never happen" then the 409s will never happen. It would have cost approximately nothing (if designed that way upfront) and inconvenienced their clients not at all.
Um, because connections over the Internet aren't 100% always on? Because packets can get lost? Because computers sometimes have to reboot?
You're assuming that the client will always receive whatever response your server finally sends, and that the client will wait indefinitely to receive a response. Neither of those things are true. So the client can be in a state where it sends a retry because it got no response and doesn't know why. And that means a retry request could come in while the first one is still being resolved--because the client had a timeout or it rebooted or something else happened that made it lose the connection state it previously had. That's the case I'm asking about.
The case of "client sends a retry with the same idempotency key" generalizes to "multiple requests come in for the same idempotency key". These can come in spread out over time (like a traditional loop), or they could come in at once. The solution is the same either way.
The problem of "how do we deal with multiple conflicting requests coming in at once" is something we have been dealing with for decades. We have databases with transactions and isolation levels. If I said in an interview "make an endpoint that inserts a value in a database and returns an error if the value is a duplicate", any competent backend web developer should be able write it without Claude's help. Concurrency is part of our life.
Whether you want to return 409 or replay the success is irrelevant to this question. You must serialize the idempotent operation on the server, because you can have multiple requests coming in simultaneously. If you put the operation in a database transaction with an appropriate isolation level, you are most of the way there.
Sure you are. You said:
"Retries will only receive 409 if the original request was successful. If the original request failed, the server performs the operation as normal on the second request. It doesn't replay failures."
I understand all that just fine; you don't need to keep trying to "reframe" it. But what you said that I just quoted above assumes, implicitly, that if you get a second request with the same idempotency key, the original request has either failed or succeeded--because you don't even address the case where neither of those things are true. I'm asking you to address that case.
If your answer is "that will never happen", I disagree, and I explained why in response to your question about why the client would send a retry if it hasn't received a response to the original request. You could answer, I guess, that you still think that would never happen--and I would still disagree. But at least that would be an answer. So far all you've done is "reframe" something that I already understand and wasn't asking about.
Whether or not a prior request exists in the system in processed or unprocessed state should not matter in a properly implemented idempotent system, the whole point is that one and only one is processed, and all replicas indicate that they are such.
What you do inside of your boundary to implement that idempotent contract need not be part of the contract and the decision of what primitives to use (locking, content-based addressing etc) are mainly just a question of implementation constraints.
I'm not sure what you mean by "in flight". The case I'm asking about is where the original request was received by the server and is being processed--and then a second request comes in with the same idempotency key. The original request has not succeeded, and has not failed--it's still in process. What response does the second request get? I do not see an answer to that question anywhere in this thread.
Here's a typical example, assuming serializable isolation in a database that uses optimistic concurrency.
* Two simultaneous requests come in to create a payment.
* The requests provide an idempotency key that is expected to be unique (possibly scoped to a tenant).
* The first request starts a transaction and starts processing, everything looks good - no dups.
* The second request starts a transaction and starts processing, everything looks good - no dups.
* The first one commits and returns success.
* The second tries to commit, but a conflict is detected (the first txn committed first). Typically this causes the second transaction to retry.
* On retry, the second transaction detects the duplicate.
The only question here is what happens when the second transaction fails? The Stripe model is "look up the original response and hand that back to the client". An equally valid and much easier to implement solution is "return a response that tells the client that there was a conflict".
Both solutions offer "create payment" as an idempotent operation.
So when the second request comes in, even though it has the same idempotency key as the first request, the server doesn't check to see if there's already a request received with that idempotency key?
That would seem to defeat the whole purpose of idempotency keys.
> On retry, the second transaction detects the duplicate.
So at this point, the second request would return a 409 code (or something like that) to the client?
With optimistic concurrency models, collisions are only detected at commit time. Two transactions can simultaneously update the same data; each update will "succeed"; when they try to commit, only the first one will succeed. The second one will fail with a code that indicates a collision. Standard practice is to just retry the transaction.
In serializable isolation, every transaction sees the state of the database frozen in time at the start of the transaction. They don't see each other's writes (that would be "read committed"). So if you have two transactions simultaneously which do "check if value XYZ exists; if it doesn't exist, insert it" they will both run the insert. The collision will only be detected when the second transaction tries to commit.
There are many other ways to implement this, but this is a pretty common approach.
>> On retry, the second transaction detects the duplicate.
> So at this point, the second request would return a 409 code (or something like that) to the client?
Yes. Stripe's approach is not fundamentally different; they just lookup the original request and return that response body instead of returning an error. It's more work for the server side engineers (and has a bunch of complex but obscure failure modes) but all the underlying database behavior is the same.
Sure, I get that. What I don't get is why you would be using idempotency keys as part of the implementation if you're going to go ahead and start a second transaction when you get a duplicate request, and not even check the idempotency key, and let your database tell you you've got a duplicate when you try to commit the second transaction. This subthread is specifically about implementations that use idempotency keys, since that's what the article is about.
It's not the default (read committed is) and I never saw serializable being set in actual production systems. You can do it, but then you have to be able to retry all of your transactions, including read.
What if the task you do take 5 minutes? 30 minutes? 10 hours? Do you create long transaction, blocking all reads?
It's not the common mode of deployment, but it's definitely in prod use.
> You can do it, but then you have to be able to retry all of your transactions, including read.
Pure read transactions shouldn't need to be retried in postgres due to serialization errors. You need to have read-write dependencies for that.
That's not to say that effectively read only transactions aren't affected by serializable, you do need to record the necessary metadata for the serialization logic to work.
FWIW, if you know your transaction is read only and long running, you can start a transaction with START TRANSACTION READ ONLY DEFERRABLE, which makes the start transaction slower, but then does not need to do any work related to serializable while the transaction is running.
Every major prod system I've worked on in the last 15 years ran in serializable, including my current charge which processes tens of billions of dollars annually. YMMV but this is quite common in serious production systems. Google's Spanner only runs in serializable.
It doesn't matter though. I could write the sequence out with a SELECT FOR UPDATE and the second request will block instead of retry. The client experience is the same; the "second" request blocks. @pdonis wanted an example so I picked one.
Idempotency keys are themselves the solution you're looking for. If they don't work concurrently, they aren't idempotency keys. Your response in races or duplicates doesn't inherently matter in that sense, pick whatever semantics make sense for your system.
No, I'm asking one question, which doesn't seem to be summarized by your summary.
The situation is that your server has received two requests with the same idempotency key. For the first request, one of three things could be true: it could have succeeded, it could have failed, or it could still be in process.
The original post I responded to said what response the second request gets if the first request succeeded and if it failed. But it didn't say what response the second request gets if the first request is still in process on the server--so it hasn't succeeded and it hasn't failed. I do not see an answer to that anywhere in this thread.
So yeah, you can do basically anything that isn't inconsistent. Success, fail, delay, don't respond until timeout, all are valid as long as you don't double-apply. Most concurrent systems are like this in some way, because all successes can become errors, and all responses might never arrive. It has nothing really to do with idempotency.
But your follow up responses here are making me rethink. Now you have to have all these special cases where the original request is still in process. I think or assertion of "99% are simple POST operations" is bullshit. For the times where idempotency is hard and really matters, often times you're calling a third party API, like a payment processing API.
I would think a better approach would be to always return a 409 on a subsequent request, regardless of whether it passed or failed, and then have a separate standard API that lets you get the result of any request by its idempotency key.
I.e. idempotent DELETE with proper protocol behavior requires that one request see the 200 OK or 204 No Content and the other sees 404 Not Found, because the delete has already happened. It would be misleading to say 200 OK to both, because that answer means the resource was there when the request arrived.
Honestly, the whole HTTP resource model has a different conceptual backing for state management than the independently developed "idempotence" concepts in distributed systems. Those non-HTTP concepts came from more message-based rather than resource-based architectural assumptions.
The cleanest mapping in the spirit of HTTP would be that you do multiple round trips. A POST creates a new idempotence context, a bit like "start a transaction". The new URI is the key for coordinating state change and allowing restart/recovery.
As I remember it, the idea of idempotence keys in headers really came from the SOAP RPC mindset. It's kind of funny to see it persisting in some hybrid SOAP + REST mental model.
I think that gave me "Enterprise Java Beans PTSD". I.e. an over-engineered solution that adds complexity for both the client and server in the name of some sort of "protocol purity".
People bolted on idempotent semantics onto HTTP because it wasn't provided natively by the protocol, so I don't think it makes sense to go through some hoop-jumping gymnastics for the sake of conforming to a spec that doesn't describe the necessary semantics in the first place.
When I let myself ruminate, it irks me that we all let HTTP become the defacto "internet protocol" just because of firewalls. Because there was a cargo cult idea that HTTP is benign and so one of few ports allowed almost everywhere, we do stupid contortions to squeeze every protocol through an HTTP tunnel.
These short-sighted acts of laziness accumulate into HTTP everywhere. And of course, the firewall is nearly pointless when "everything" is going through that one hole anyway.
This isn't a special case, and it's the same problem if you want to replay the original response on conflict. If the original request isn't complete, what are you going to replay?
Who says you have to replay? If you get a second request with the same idempotency key, and the original request is still in process, why not just send the client a response that says so?
Long running transactions create all sorts of problems, so transactions are generally expected to be short. The actual work behind "create payment" or "create order" is generally fairly trivial - more or less insert a row in a table. There's no good reason to make the API complicated... you either "win" at concurrency or you lose, and the difference is generally sub-millisecond. The only meaningful thing you need to communicate to the client is "you're done" (for both the win and lose cases) or "you need to try again" (for the "something unexpected went wrong" case).
Complicated workflows can certainly have multiple steps, with "fetch the current status" calls in between. But somewhere near the beginning of every complicated workflow there will be a call to "create workflow" and it will need to have sort of mechanism which allows clients to call it idempotently. Otherwise you end up with multiple starts.
I've literally received duplicate products in the mail because of this kind of problem. I've also sent multiple products in the mail because services I relied on didn't offer the necessary idempotency mechanisms.
It's generally insert a row in someone else's table, over the wire, 50ms+ away. They might not even be using an RDBMS.
Devs are too scared to be nice (ie not return errors) to clients when they misbehave.