If the request never goes through, then I can always get the customer to try again, but if it gets charged more than once, then it's a huge headache, and our customers get extremely angry and lose confidence that we know what we're doing.
The reasoning at the protocol level then goes, "If I have never seen this nonce, then the transaction has not been authorized" (very durable, easy to implement), and "A request seen multiple times will only be recorded once" (also very durable and easy to implement).
But if the message infrastructure tries to guarantee at-most-once, and the user writes their protocol, "if I see this authorization I should do it, because the protocol will only send it to me one time." Then there will come a time in that protocols future where something blew up the invariants in the message infrastructure and the message arrives twice, and the wrong thing happens on the credit card because the protocol trusts it to do the right thing.
Back in the RPC wars people were arguing that it was so much "simpler" and more "transparent" if a remote procedure call had the exact same semantics as a local procedure call. And that is true it would be, if you could manage that similarity, except it has been proven over and over again that network messaging systems can't completely get it correct.
- Each worker has to hook to a database where a record keep the state of the transaction: ready, processing, success or failure. With a good old SELECT FOR UPDATE a transaction will never be done more than once.
- Timestamps and alerts ensure that if the worker fails dramatically (leaving the task as "processing"), a human will check and update the task accordingly.
It has proven pretty resilient so far... but at very small scales.
An at-most-once policy doesn't save you from that, but an at-least-once policy would force you to write an idempotent handler that may be able to recover from failure (via the PP API) or safely make progress (with a bit of transactional state).
Agreed about idempotent processes in the "consumer" processes. But I generally prefer marking an end user session in a persistent store before propagating the request for payment (effectively acting as a sticky current transaction ID), and making synchronous requests down the chain versus generating a transaction ID and doing a bunch of async message queuing.
It's just easier to know when something didn't work and display the appropriate user feedback. But your point is valid.