Exactly-Once Delivery May Not Be What You Want

> One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder.

This is a wisdom whispered 3 decades ago. The end-to-end argument haunts us again [1]!

[1] END-TO-END ARGUMENTS IN SYSTEM DESIGN http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

The point about unique IDs is well taken, but there's a cost to tracking these unique IDs too.

For example, if you generate hundreds of message a second, you quickly end up having to track millions of unique ids. Then you have to manage the storage of these IDs somehow, so you end up implementing a system to wipe out any ID older than a certain time, and so now you not only have track the IDs you've seen, but when you've seen them.

That said, this is probably better than making an incredibly complex message queue system just to support exactly once delivery.

As a maintainer of logging middleware (https://www.fluentd.org) I am asked all the time whether we have "exactly once" semantics. I try to be honest and say we err on the side of "at least once" message delivery, but even then, there are edge cases where messages can be lost.

It really vexes me to see how a lot of distributed systems projects claim to support "exactly once", and it makes me happy to see a grounded, intelligent summary like this one.

> One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder. The other is that simple, practical, solutions like unique IDs can make really hard problems much easier, and allow us to build and ship real systems that work in predictable ways.

These assertions that I tend to follow most of the time. I keep warning my team members to never assume that a request for payment/fulfillment/some-business-function is going to arrive exactly once. If they build systems assuming that underlying scaffolding is going to behave in certain way then they won't be designing for failure. In any distributed system it's very important to define correctness from the end user's perspective. And then design for failure ensuring that the correctness guarantees are not violated.

This is a nice article, but the title strays into 'Fox and the Grapes' territory. Exactly-once delivery is pretty much always what you want -- in contexts where all manner of delivery guarantees are possible, from Java's concurrent queues to Go's channels, pretty much everyone delivers a message precisely once.

Of course, you can't always get what you want, and in a distributed system exactly-once delivery is not possible in general. But in the cases where it is possible, it's often worthwhile: it's just a whole lot easier to reason about than a system where messages can be arbitrarily re-delivered.

Might be a straight-forward application of Buridan's Principle: http://research.microsoft.com/en-us/um/people/lamport/pubs/b...

Obviously, exactly-once delivery is a side effect of possible overproduction and then filtering for uniquenessz

Some people might also say that creativity is a side effect of brainstorming and idea selection. But it's more complex than that - over time our brains work out heuristics and figure out patterns that work well, and use that as a language to construct other things. That's why people who have been doing something for years can do it on a much higher level.

This is a fantastic summary of the challenge of distributed execution. I also really enjoyed the quote "there is always room to slot in another turtle" I am definitely going to keep that one handy.