Falsehoods programmers believe about TCP
lwn.net
lwn.net
1. You're wrong
2. Okay, you're right
3. Okay maybe you're right or wrong but certainly not both
“Falsehoods programmers believe…” articles are designed to make you THINK about problematic assumptions. They are not like the 10 commandments and they are not decrees of absolute truth.
But I do agree that some of the later entries have felt a little lazy.
[1]: https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/
[2]: https://shinesolutions.com/2018/01/08/falsehoods-programmers-believe-about-names-with-examples/1. That said items are falsehoods in the first place.
2. That said items are necessarily interesting or noteworthy.
3. That a list is necessarily the best format to present said items.
4. That they may speak for the involved parties beliefs.
1-4. Yes we know about the 2 generals problem. And yes we know what "reliable" means in this context. 5-6. This is just stupid. 7. Obviously not true. Nobody thinks this. 8-9. The reasons for and flaws of Nagle's algorithm are well known. 10. This isn't even true. Most of the time you don't need to care about it. That's the whole point of abstraction. You need to care about it if you are doing extensive performance optimisation, but usually you aren't. 11. Again untrue. You can think of TCP as a two way pipe. Again that's the whole point of abstraction. 12. Not sure exactly what they're trying to say here but again it's very well known that TCP and UDP are pretty much the only protocols that are likely to work on the internet. 13. Ditto. We all know why so many protocols are "over HTTPS", e.g. DoH. 14. This isn't a technical point. 15. Dunno what this is talking about but I'm guessing it's along the lines of "a byte is 8 bits", i.e. it is actually true in the modern world.
It’s like an engineer who says “how does a screen show black” and then says “nope” to every response. It’s maybe a way to make people think, but beyond that the negativity and grandstanding of it is ultimately a turn off for many receivers which eventually either then has them bully others this way or deters them from the field, depending on how it affects them. There are far better teaching methods that work better for everyone and teach faster and result in higher accuracy and retention.
The reset of the writing just laments the pain of using products that make incorrect assumptions – continuing in same lamenting from the quoted segment he includes. It almost has nothing to do with TCP at all, so it is not clear where the parent comment here got the idea that it was trying to teach something about it.
Some displays are implemented with dual-eyelid technology for the blackest of blacks. Naturally, like all genius engineering, we see this in nature: cats.
In case of e-paper, black pigment is attracted to the outer part of the screen.
What I have said is something to the effect that if TCP isn't reliable over a given path, there's not a whole lot I can do about it as an application engineer short of making my own ad hoc, informally-specified, bug-ridden, slow implementation of half of TCP inside my own app, which I'm not going to do.
> 14. Weird networks that are not transparent to standard protocols are an aberration. I can safely ignore them.
I certainly can and will. If you wanna run an RFC 2549 network, I'm going to spend approximately 0 seconds making my app support it. If you want to do something weird, you make it work. I'm going to optimize for the other 99.99999% of customers.
https://infiniteundo.com/post/25326999628/falsehoods-program...
Maybe he’s referring to the fact that some last names are two words (e.g. Marine Le Pen), but I don’t think that’s very common…
Anyway, it could be anything, so I wish he’d said!
Anecdotal, but the only people I've met in the US with more than one middle name are people who originally came from another country.
Although, I wonder if maybe that is enforced by the fact tha legal forms and similar typically assume you only have first, last, and optionally a (single) middle name.
You don't have to defend the "original" post just because it was patio11's. This idea was awful and stupid from the very beginning, and every new post of this "series" just repeats the offence.
Besides, AFAIK he only changed his stage name, not his legal name.
China (and Japan, and to a certain extent Korea and Viet Nam) exists? https://en.wikipedia.org/wiki/Naming_laws_in_China#Notable_c...
I'm pretty sure that patio11, having spent his life in Japan, would know that technology like SING glyphlets exists because of this exact issue.
(and before you answer "what are you talking about, it is in Unicode?", these characters are literally added after the relevant issue surfaced, and some characters like 𱅒 (U+31152) are recent additions that don't even render properly)
For any given fixed size that people use in practice, there is a name that does not fit. This is saying "use a variable-length field for names, because there are always edge-cases".
> Jullien was born in Sisteron, Alpes-de-Haute-Provence, and was baptised Louis George Maurice Adolphe Roche Albert Abel Antonio Alexandre Noë Jean Lucien Daniel Eugène Joseph-le-brun Joseph-Barême Thomas Thomas Thomas-Thomas Pierre Arbon Pierre-Maurel Barthélemi Artus Alphonse Bertrand Dieudonné Emanuel Josué Vincent Luc Michel Jules-de-la-plane Jules-Bazin Julio César Jullien. His father was Antonio Jullien, a violinist. The explanation of his unusual number of names is that when the time came for the baby to be baptised, his father had been invited to play at a concert given by the Sisteron Philharmonic Society, and considered it only polite to ask one of the members of the orchestra to be godfather: but since every member wished to be considered for the privilege, he was christened with the names of all thirty-six members of the society.
They have not written a "falsehoods programmers believe" article. They have proposed that one ought to be written and have given a starting point for what it might cover.
They offered their list to "get the ball rolling", confirming that they don't see it as a finished product.
They sent it to other readers of the same forum, who might be expected to have more knowledge of this topic, not to whoever runs across it on the front page of HN.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
https://beej.us/guide/bgnet/ -- Covers what abstractions the OS provides for network programming and the guarantees that are possible.
https://www.madwizard.org/programming/tutorials/ - This is the very first ever good tutorial I read on socket programming. It's OG winsock. Introduces network programming from the most basic level. Aimed at C.
When you understand these guides you'll learn that how you structure your entire programs networking depends on whether you want to use blocking or non-blocking sockets. If you go with blocking you'll probably be using threads or processes. Otherwise you can't do any other work. With non-blocking it will be more about polling sockets and eventually you might end up with something resembling an event loop.
Until you come towards to the current approach to networking which is mostly async await -- an event loop works with non-blocking sockets, watches them for changes, and passes data from them to event handlers. There's a lot more that can be done on sockets to effect things like how data is flushed, how TCP errors are handled, and so on, but its a good start.
> 5. There is a such thing as a TCP packet
> 6. There is no such thing as a TCP packet
I don't understand this at all. Either the concept of a TCP packet exists, or the concept does not exist. Even it's not being used in certain scenarios, I don't see how you can argue that "there's no such thing" any of the time. This might just be me misunderstanding whatever point they're trying to make, but I don't remember ever having such philosophical confusion from anything in any other "falsehoods programmers believe about..." article before.
I think the thing most related to that that I see people thinking is that send(2) & recv(2) calls translate 1:1 with packets send/recv. I.e., that they don't understand that the interface TCP exposes to applications is a byte stream. Which then results in things like thinking recv(2) will receive a complete "message" for some definition of message in the application protocol (i.e., the mistake belief that fragmentation won't happen).
My guess is that this is talking about systems in the middle of the network, changing (for example) their sizes by combining and splitting packets to fit through various transits.
In practice such situations can arise in one of two cases:
1. some non-sense creeped in
2. logic is applied to a self-contradictory set of axioms and definitions.
(1) is not very interesting, but (2) happens frequently enough because people often do not try to formalize their definitions and axioms. As a consequence they are using some vague concepts and their statements are true in some cases but not in others.
With all that said, I can propose the way how this logical non-sense could be right. (NB. I don't know if it applies to TCP, I'm just thinking generally, and just as an example to all that abstract words above) The notion of "existence" of the mistaken programmer can be wrong. If we accept their definition of existence, then TCP packets doesn't exist, but they exist in some other sense.
Either way, this stuck out to me because usually these type of lists have very simple, understandable statements that just happen to surprise some people by not being true; in this case, the statements themselves are confusing, so I can't really say for certain whether I believe them or not because I don't even know what truth they're asserting to try to negate it.
I would classify these statements as koans, and further observe that one of the defining characteristics of these lists is that they are wry. With that in mind the couplet is evidently presented in cheek to illustrate the need for context, since this is the pathway to resolve the overt contradiction without getting stuck in a rigid thinking trap.
https://networkengineering.stackexchange.com/questions/50083...
In particular, fragmentation by intermediate routers means that the server and receiver may disagree about the frame and packet boundaries. TCP is expected to make a "reliable" pipe-like service out of whatever happens, and the application layer doesn't have (shouldn't need?) visibility into that process.
Fine. It's not wrong about absolutely everything all the time. It isn't bereft of all truth. It's just something that is of net negative value. I see no value in insisting on trying to "rescue" a net-negative value model of the world.
I suppose you could say ultimately I agree with you though. The OSI model isn't useless. It's worse than useless. You're better off trying to understand networking from basic first principles than through the lens it provides.
I mean in practice it's so broken, imprecise and messy to a point it's often more misleading then helpful and IMHO should have been replaced in teaching with something better well over a decade ago.
And to be clear I'm not saying it's bad because some small implementation details don't fit, it's bad in it's job of being a high level abstraction where you ignore many implementation details.
If I recall correctly, the formal term for the all-encompassing category is the "datagram".
It's like who-vs-whom. Prescriptivists will try to tilt that windmill forever, descriptivists observe that most people just don't care.
Because the software abstraction is a stream of bytes; and it's up to the application to decide where the "packets" begin and end.
For example, I might write to a TCP socket: 100 bytes, 50 bytes, and then 125 bytes.
BUT, the receiver could get: A single event with 275 bytes. Or it could get an event with 75 bytes and then an event with 200 bytes. Or it could get 11 events of 25 bytes.
> 5. There is a such thing as a TCP packet
This one I struggle with. I think the author is talking about connection set up, acking, and connection teardown.
TCP is a protocol that most certainly has packets ("segments").
You are taking about a SOCK_STREAM-like programming interface, which can be used to create TCP connections.
I work with DNS a lot and when a protocol which is datagram-oriented is translated to a stream-oriented medium and somebody wants to potentially handle multiple requests in that stream because "efficiency"... it's so important... they need a way to distinguish those embedded datagrams: "we should do it they way they did in HTTP, with a Content-Length: header, yea THAT's the ticket!" I'm sure that's what they were thinking.
Then along come the 90% and when they try to process DNS requests in TCP streams it's "what's this two bytes in front of the request? I dunno, but I just skip over it" and I suppose it works well enough, because with frags they'll drop the tail of the requests because "corruption" anyways, and who on earth sends multiple requests in a single packet, amiright?
You may think I'm kidding, but here's some really clever (really!) eBPF code: https://gist.github.com/oghie/b4e3accf1f87afcb939f884723e2b4...
If you want more on this topic: http://consulting.m3047.net/dubai-letters/dnstap-vs-pcap.htm...
5.,6. are likely about how all kinds of things infer with your packages mainly merging/splitting but also potentially messing with them in other ways (and in general the package->frame mapping). So if you want to build anything which relies on TCP packages being a thing it likely will not work, at the same time you can't always ignore package size either (e.g. for flushing byte streams, higher level protocol chunking etc.). Through as long as you don't create transport protocols write a network stack or fine tune your TCP stack or similar you likely don't have to care about it. I mean you also probably shouldn't use TCP directly but something more high level in most situations (e.g. HTTP, gRPC, etc.).
1. Remote keeps sending stuff to your unplugged connection
2. You plug your ethernet cable back in
3. Your computer's TCP acknowledges the last sequence number it received for each new sequence it receives from remote
4. Remote sees duplicate ACKs for same sequence number, interprets it as packet loss and resends the stuff
If packets were sent while you were disconnected, they'll be gone, but if you're disconnected for only part of the burst, duplicate ACKing will trigger retransmits.
If you were gone for the whole burst, you'll get put right by timer based retransmits.
If you're gone for long enough, most peers will timeout on unacknowledged data (although that's not in the TCP RFC), and if there's no outstanding data, most peers eventually have some sort of periodic ping and timeout (tcp keep-alives is a reasonable fallback IMHO, if your application protocol doesn't have someything, although the default of IIRC 2 hours feels long in todays world of lots of NATs and much shorter timeouts).
It's rather irritating.
Rather than "IDK keep sending but I haven't heard ACK in a while"
It's the conceptual equivalent of your phone "helpfully hanging up" when you change the phone from one ear to the other during a call.
Later, Microsoft made it configurable:
https://learn.microsoft.com/en-us/troubleshoot/windows-serve...
Of course if the connection is active when that happens, the other end will probably not like the prolonged silence, and give up first.
edit: Point is in confirming that message is received. If you don't receive the confirmation the message was delivered at most once.
You have very limited guarantees around an arbitrarily bad partition, but this is also a detectable condition. Lots of defective systems exist, but in general non-defective systems generally guarantee "exactly once delivery or detected failure"
Other people upthread have already gone over how you can't separate delivery from message processing and how TCP's attempt to do so makes it defective (unless you layer a whole additional system on top of it rendering most of TCP's design irrelevant)
If you were trying to make a new, non-broken system on top of TCP or otherwise, allowing multiple delivery doesn't add any correctness/robustness benefits -- it just makes messages cheaper to send and receive. There is no "at most once or at least once" choice except in the pickwickian sense that if you don't require delivery or delivery confirmation you can save the effort of even trying.
[UPDATE] Apparently I need to be more explicit about this. My question is: if I can get at-least-once delivery, why can I not build an abstraction layer on the receiving node that provides the illusion of exactly-once delivery? It seems like it should be a simple matter of keeping a log of received messages, and discarding duplicates.
The difference is subtle but important in practice and specification.
Why? I'm pretty sure I really want (the illusion of) exactly-once delivery, and it seems to me that I can implement that pretty easily given at-least-once delivery. Why would I not want that?
> The difference is subtle but important
Why?
Do you know what idempotency is? This is exactly what he described.
Idempotency is important to prevent unwanted behaviour for duplicate actions. If you have "exactly-once", and accidentally execute the action twice that could cause problems.
Yes.
> This is exactly what he described.
So? Idempotency and an exactly-once delivery abstraction are not the same thing.
Idempotent processing is exactly once processing. This is true. Delivery is from the sender's point of view, not the recipient. How the recipient processes the information, idempotently or not, is not the concern of the sender.
"Cannot have exactly-once delivery" is true and is as true as not dividing by zero. Read about the two general's problem. It is, quite literally, impossible to have exactly once delivery. Like, your friend is in the other room, and you shout "someone is at the door" - how do you _know_ your friend heard you? If you shout again, you are attempting another delivery. What do you do if your friend doesn't respond? In exactly once delivery, you guarantee they heard you in the other room. In exactly once processing, you can shout a couple of times until you hear them acknowledge.
You may think that this is not material and at the end of the day, as long as one thing is processed, then who cares? Well, you have to understand how delivery can fail otherwise you will handle the failure incorrectly. Is it safe to try again? If I say "transfer money from me to you", and I don't hear back, is it safe to again say "transfer money from me to you" again? Will I be double charged?
You are not the first to say this, but so far no one has been able to explain what the difference between "delivery" and "processing" is. How do you do "delivery" (on a computer) without also doing (at least some) "processing"?
> Delivery is from the sender's point of view, not the recipient.
I don't see what difference that makes. In fact, I don't see why a "point of view" should enter into it at all. Whether a message has been "delivered" or not (whatever that actually turns out to mean) is (it seems to me) a property of the system, independent of anyone's point of view.
One of shouting at your friend. You want to make sure your friend knows someone is at the door. Two: you tell your computer to transfer money and you don't want to be doubled charged.
More in depth: you click send money on a computer. The computer connects to another computer and sends data to it over an unreliable network. Computer A sends data over the network to B, just like you shouting to your friend you think is in the other room. Data can be lost / your friend might not hear you. Usually, the other computer says "acknowledged, I got your message" - and that is how know that B is moving money or your friend is getting the door. If A never hears back, should A try again? If B gets two requests to move $25, should that be deduplicated or was there two actual requests and $50 should be moved? To know how to solve that, you have to first admit that you might get 0, 1, or multiple messages delivered to B when A wants to send 1 message.
Read the two general's problem. It is the gateway to distributed computing.
Neither of which was on point because they both ignored how exactly-once delivery can be done.
> Read the two general's problem.
The 2GP is not on point because it's about achieving consensus, not exactly-once delivery. Achieving consensus is indeed impossible with unreliable messaging, but that has nothing to do with exactly-once delivery.
Is it though? It seems like a false equivalency, even if the outcome is approximately the same?
But not 100%. At some point, a counter move on the delivery has to be stored... -somewhere-.
And sure you -can- make it very very close to EOD and for some subsets you can totally do EOD, but you are, realistically, better off with ALOD+Ack once it makes its way into a system. There's always that 'moving the counter' problem.
The upshot is, things tend to get faster, easier to code review, and simpler to test.
Pragmatically speaking, I've found devs are better able to handle ALOD+ACK than "Exactly once but because reality you might get a message that's doubled because you couldn't persist the ack".
And I'll note I'm possibly extra pedantic about this because I've had a month and a half of dealing with the fallout of people trusting low-code salesmen alongside gartner reports leading to a 'you people thought this was exactly once and it was not' sort of problem.
Why not?
> At some point, a counter move on the delivery has to be stored... -somewhere-.
What is a "counter move on the delivery"?
https://bravenewgeek.com/you-cannot-have-exactly-once-delive...
This is why people like me insist it's important to understand that you can not have exactly-once delivery. There is no library that can make that just go away, they can only shuffle around exactly where the lumps under the carpet live, and if one programs with the mistaken idea that these libraries really do solve "exactly once" delivery, one will get into deep trouble. Possibly the "my architecture is fundamentally broken and can't be rescued" sort of trouble.
Why do I need to understand that? Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery?
> There is no library that can make that just go away
Well, this is the thing that I dispute. I believe that there is a library I can write to make it go away if I have at-least-once delivery. In fact, I claim that writing such a library is an elementary exercise. The TCP protocol is an existence proof. Where is the flaw in my reasoning?
You can do that. You can implement a video conferencing system ontop of TCP, and it will even work, technically. It will just have terrible performance characteristics that you'll never be able to fix. You might even call it fundamentally broken.
Like you said, you can simulate it with exactly once _processing_. But to do that, you have to know to do that.
Others are saying "you can't divide by zero" and you are saying "yeah, but if I detect a zero and the do something different, then it is the same thing." No, knowing you have to do something is the very point of acknowledging you can't divide by zero.
Because you can't have exactly once delivery, you have to deal with it. One trick is duplicate checks or idempotent writes. This gives exactly once processing. This also takes additional overhead and why audio and video stream processing doesn't typically do the additional checks.
I have fixed many bugs written by people who believe the network is reliable. I even hired one when during the interview and we talked about this kind of issue that they realized why they were getting duplicated writes reading from sqs or sns at $existing_job. They were green but smart and she became a real asset to the team.
> you can simulate it with exactly once _processing_
What exactly do you think is the difference between "simulating" exactly-once delivery and actually having exactly-once delivery? What do you think "delivery" means in the context of a computer network?
Here's another useful article: https://blog.bulloak.io/post/20200917-the-impossibility-of-e...
TCP does not solve the two generals problem. TCP gets around this limitation by requiring only one ACK.
https://en.wikipedia.org/wiki/Two_Generals%27_Problem#Engine...
https://www.scaler.com/topics/computer-network/two-generals-...
I think there is a lot of literature out there if you're really interested in understanding and I'm happy to provide more links if you'd like.
Particularly:
> The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or network unreliability).
> The recipient is forced to send the acknowledgement only after the message is either processed (or persisted for processing) because an acknowledgement before processing would not work: if the recipient exhibits a fault before processing that would cause the loss of the message.
> In the case that that acknowledgement is lost, the sender can’t know (due to lack of insight in the recipient’s state) whether the recipient failed before scheduling the message for processing (in essence losing the message) or if the recipient is just running a bit slow, or if the acknowledgement message was lost. Now, if the sender decides to re-deliver, then the recipient may end up receiving the message twice if the acknowledgement was dropped (for example). On the other hand, if the sender decides to not re-deliver, then the recipient may end up not processing the message at all if the issue was that the message wasn’t scheduled for processing.
"Let’s assume that a protocol exists which guarantees that a recipient receives a message from the sender once and only once. Such a protocol could then solve the two generals problem! Representing the time of the attack as the message, the first general (the sender) would only need to adhere to the protocol for the second general (recipient) to have received the attack time exactly one time. However, since we know that this is not possible, we also know that exactly once is not possible."
The 2GP is not just the first general knowing that the second general received the message. The 2GP is the problem of achieving common knowledge between the two generals, i.e. it's not just that G1 needs to know that G2 got the message, it is that G2 needs to know that G1 knows that G2 got the message, and G1 needs to know that G2 knows that G1 knows that G2 got the message, and so on.
Exactly-once delivery is possible. The only thing that is not possible is for the sender to know when the message has been received so that no duplicates are sent. But exactly-once delivery is not only possible, it's trivial. All you need to do is discard duplicates at the receiver.
...yes, which would be exactly once _processing_ but not exactly once _delivery_.
Unless you're wanting to redefine "exactly once delivery" to mean "at least once delivery but I'm calling it exactly once because I have a strategy to cope with duplicate messages"
What exactly is the difference? What counts as "delivery"? How do you do "delivery" (on a computer) without doing at least some "processing"?
Communication is when two actors exchange a message. Communication is generally done over an unreliable medium because in practice there is no way to communicate without the potential of failure.
1. Exactly once communication between two actors over an unreliable medium is impossible. At the very least you have to account for the possibility of failure of the medium, so you might need to re-send messages.
2. At least once communication between two actors is possible -- just re-send a message until the receive acknowledges the message.
3. Because a message might be re-sent, the receiver must be able to cope with duplicate messages. This is what you're describing. This might be done by making message processing idempotent or tracking which messages you've seen before. In either case, you have achieve exactly once processing. That is, if a receiver is given the same message multiple times, only the first receive of the messages changes the state of the receiver.
---
> But exactly-once delivery is not only possible, it's trivial.
Considering that many in the field consider this problem to be impossible (or, at best, extremely difficult, e.g. https://www.confluent.io/blog/exactly-once-semantics-are-pos...), this should be a huge red flag to yourself that you're missing something. Everyone has blind spots and that's okay, but hopefully you understand that there's a pretty big mismatch here.
Alternatively, it's possible that this problem _really_ is trivial and you have some unique insight which means there's a great opportunity for you to write a paper or blog post.
Yep. But on the other hand, 1) I have a Ph.D. in CS, and 2) I have yet to see anyone in this thread actually produce a reference to a reliable source to back up the assertion that exactly-one delivery is impossible. Indeed, the one reference you provided has a headline "Exactly-Once Semantics Are Possible" so you are actually supporting my position here.
Finally, I will point out something that should be obvious but apparently isn't: "exchanging a message" between computers over a network is a metaphor. There is not anything that is actually exchanged, no material transferred from one computer to another. There is only information sent in the form of electrical signals which results in state changes in the receiving system, so there is no clean boundary between "communication" and "what the receiver does with a message". Receiving a message in the context of a computer network is necessarily "doing something" with that message. There is no other way to "receive" a message.
There is no guarantee that the application reads from that buffer (e.g. the process could crash), so the client on the other end believes that the application has received the message even though it hasn't.
The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.
What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
> TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data.
True. Why do you think that matters?
> There is no guarantee that the application reads from that buffer
So? What does that have to do with exactly-once delivery? Even if the application does read the data, there's no guarantee that it does anything with it afterwards.
> The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.
OK, if that's how you're defining your terms, I agree that you cannot have exactly-once delivery. But that makes it a vacuous observation. You can always get at-most-once delivery by disconnecting the network entirely. That provides a 100% guarantee that no message will be delivered more than once (because no message will ever be delivered at all). But that doesn't seem very useful.
The difference between "processing" and "delivery" relates to "network capacity." Process handling wastes capacity in favor of latency. Delivery handling increases latency in favor of capacity.
Systems which have "exactly once" delivery typically do so with "send/receive" and "release/delete" message pairs. You need additional round trips to actually accomplish this at the "delivery" layer.
Yes, I know, which makes it all the more bizarre that people are claiming that this is impossible.
> Systems which have "exactly once" delivery typically do so...
Ah, so exactly-once delivery is possible after all?
So this hypothetical middleman will receive a packet, check that it's not a duplicate, and forward it to the recipient it's proxying for. How will it know that the recipient has actually received it? If the receiver doesn't ack the message in some way, which causes your abstraction to re-transmit the message again, then it exhibits 'at least once' behavior. If it the abstraction only ever forwards the message along once and doesn't care whether the recipient acknowledged it or not, then it exhibits 'at most once' behavior.
As a more concise answer - 'exactly once' delivery is impossible because you can't know if the recipient actually got the message. If you assume a perfect communication channel, then I agree the problem is trivial, but I challenge you to find such a channel! Even on the same machine, interprocess communication can fail in all sorts of fun ways.
That's not how I would implement exactly-once on top of at-lest-once. I would do it at the recipient, not at the intermediate nodes.
> 'exactly once' delivery is impossible because you can't know if the recipient actually got the message
But the recipient can know.
There's still communication happening there, and it can still fail. Buffers fill, processes pause for arbitrary delays which exceed timeouts, etc. Your assumptions based on your model are correct, but your model doesn't include that communication.
But all models have some level of detail they care about, and assuming the computer always works is a perfectly valid model for plenty of cases. It's just not all of them. You'll be able to create real-world cases where this abstraction is faced with a choice of whether to retry or not, and at that moment it will be unable to deliver exactly once.
My claim is not that you can provide exactly-once delivery unconditionally. My claim is that if you can provide at-least-once delivery then you can turn that into exactly-once delivery. The word "delivery" is not rigorously defined, but IMO any reasonable definition necessarily entails a certain level of reliability at the receiving node.
The meaningful distinction is that something on the recipient needs to be idempotent because the message might get received twice. The application can be oblivious to this, so long as you assume that channel to be perfect.
People on the Internet won't like you calling it 'exactly once delivery' because it's not exactly once - it's an idempotent at-least-once. Which is great! But the statement of the at-least/at-most problem is making a decision to re-try. There's no middle ground, I either have to retry or not. People won't like a claim that 'exactly once' delivery is possible, because it isn't, it's just moving the at-least-once-ness to somewhere else.
That depends entirely on what you mean by "it".
Messages can get lost. So if you want guaranteed delivery, you sometimes have to re-transmit the same message, and so you might end up with the same message being delivered more than once. But it is trivial to put an abstraction layer on top of that to discard the duplicates and make it appear to the application as if every message is received exactly once.
The whole thing is a tempest in a teapot.
It seems like the answer is in the first part, the "check that it's not a duplicate".
Implement at-least-once but with a unique token to identify the request, and the receiver sends back acknowledgement with that token every time it receives the original message, but only hands it off for processing the first time. Stuff this behind library/API so it's hidden to the user and the application code doesn't have to handle it, and... isn't that it?
Everything in a digital system can fail, but by convention (and because it's not so far from the actual truth) some parts are assumed to be 100% reliable when modeling them. If you don't make this assumption, you can't guarantee anything.
That would be like:
Alice: "Honey, where did you put the keys?"
Bob: "They're up on the counter."
(The next day...)
Alice: "Honey, where did you put the keys?"
Bob: (nothing, I already received this message, it could have echoed off the walls from yesterday)
What you need is for all sent messages to have unique IDs that will never repeat, and then log those. That's known as an idempotency token.
But even then, logging all those UUIDs forever is probably not a good idea for disk usage. At some point you'll have to trash old message logs and hope you don't have a rogue network router retransmitting six month old messages or something.
[0] Or the moral equivalent of messages, e.g. HTTP POST requests
So your problem is not really a problem because you yourself present the solution. The real problem is:
> But even then, logging all those UUIDs forever
But you don't need to log them all forever. Just make the UUIDs sequential, and all you need then is to keep track of the smallest id that has not yet been received. (You can be more efficient by storing more state, but it's not necessary. Remember, we're assuming at-least-once deliver here, so you can always force retransmission by not acknowledging receipt.)
Your network won't have "exactly once" message transfer happening on it (it will internally be "at least once" for certain packets, but only small ones) and administering it will be very different than administering an Ethernet network, but network protocols absolutely can be designed to give exactly-once delivery to your software.
The real reason most people outside of HPC don't do this is that exactly-once at the network layer is not that useful for most web stuff. You're going to have a higher layer that will drop stuff and retry anyway, so you might as well push the problem up the stack.
Additionally, I'm sure they're aware that HTTP over TLS has encrypted data frames, which would be unreceivable in a lot of cases if these situations arose a bunch. And considering how much of the modern Internet is built on this paradigm, I think that many of these points are rare and probably extremely pedantic.
This is coming from someone who agrees with much of the nuance implied (but not explained!) by the post.
All great technical writing (which I assume these clickbait articles are at least attempting to be) is written with mutual discovery and deeper understanding in mind, and if you leave no actual explanation in the post, you can't really achieve either of those.
Neither TCP nor Ethernet provide for forward error correction. Ethernet frames include a 32-bit CRC while TCP segments use the so called "internet checksum".
The real question is, why this should be a problem that TCP must solve? TCP gives you a bidirectional waterflow-like pipe, and that's enough for you to create many useful applications. TCP never provided guarantee for correct delivery, that's your job.
For example, if a HTTP request is interrupted before the respond is received, the sender should assume the request never reach the server and try again with a new connection, while the server should mitigate duplicated requests (reject or return a successful code).
Well, maybe that's the point of the article, because many web pages gets confused if you send duplicated requests to them.
“Mostly” because you still care about bandwidth limits and packet RPS limits and latency of course.
If you've got dirty optics between two switches, now you're getting packet loss and TCP rears its head. Hopefully it's not an issue now, but diagnosing microbursting[1] was lots of fun, and really wigs TCP out. I've also run into 'fabric congestion'. My true favorite though is when you've got 2x aggregation on servers, and 4x aggregation for top of rack switches to spine switches, so there's 8 paths in each direction between two servers in adjacent racks, and only one path (sometimes in only one direction) is only running at 99.9%. That's a real PITA to track down unless you have visibility into switching metrics.
[1] https://en.m.wikipedia.org/wiki/Micro-bursting_(networking)
Given how generally reliable switches are, I was inclined to believe that a misconfiguration or flaky network cable on one switch was more likely to cause a downtime (or significant degradation) than an outright switch failure, so adding another switch was doubling the chances of trouble and, as you note, making it harder to troubleshoot.
Otoh, if you ever do any scheduled maintenance on your switches (which is likely if they're doing anything fancy), having properly setup redundancy means you can announce a likely brief loss of redundancy, rather than a likely brief full loss of connectivity. If you have the right knobs, you can gracefully fail out the switch under maintenance and everything goes smoothly. Of course, sometimes you reboot the redundant switch and it confuses the other one and servers lose connectivity anyway.
Having done Akka.NET Remote/Cluster setups in prod that survived multiple 'new to the org' categories of DC Failures at their level of scale/capacity [0] there's a lot to account for if you want to keep everything happy and visible [1][2][3]
[0] - Cut fiber between DCs, Rack failures due to IO-ish type issues, bad switches... at least 2 out of 3.
[1] - The upshot was we were able to survive all of the scenarios in at worst a degraded state, Once or twice we needed a restart.
[2] - We also had enough metrics going on that we could detect DC/server outages about as quickly as whoever actually was monitoring the failing subsystem.
[3] - But here's the funny rub. An APM tool was the Achilles heel for both our Akka Links, as well as our SQLServer connections. Once they installed an 'agent' we more frequently had to do a 'full cycle' to clean things up after an outage, or even an MSSQL Server reboot. After I left the shop I got confirmation that yes, the APM module was the problem.
Yeah, my Erlang clustering experience was that we (the customer) were the monitoring system for the DC/managed hosting provider. Although, by the time we left there, they would have outage notifications before we put in tickets.
The problem: you're on a subway train and you send a message as it departs a station. The request does get to the server, but by the time the response arrives, the train is already in the tunnel and you don't have a signal any more. So the client thinks that the message failed to send, but it was, in fact, sent successfully. The client would retry when it's back online, and would send another copy of that message.
The solution was to send a client-generated "random ID" with each request. I much later learned that this is conventionally called an "idempotency token". This worked, except there was now another problem: you sometimes receive your own message over the long-polling thing before the response to the request that sent it. You don't know for sure whether it's the message you just sent, or something else sent by a different client on the same account, because you don't know the ID of your message yet. This was solved by me delaying the processing of outgoing messages on the client side until all outstanding messages are fully sent and their IDs are known.
Telegram solved this much more elegantly: when the client reconnects to the server, the server sends it all the responses that were not acknowledged during the previous connection. MTProto has its own acknowledgement mechanism in addition to TCP's.
So yeah, instant messaging seems trivial at the first glance, but it turns out that TCP is a leaky enough abstraction that you need to somehow plug those leaks at the application level.
Wouldn't the client know the ID if they randomly generated it? (honest question)
Recently a new rate limiter for TCP went by that was so terribly, terribly broken, and I cannot help but imagine that most of the containers of the world suffer from Bufferbloat in general.
In what way is that a falsehood?
but you can get round that in a lot of cases by just having a load of TCP connections in parallel.
TCP is cheap and well optimised, especially if you are keeping a bunch of connections open. (opening can be expensive)
so if you have a high latency connection, or a bit of packet loss, and you want to reach line speed without having to figure out cornercases with UDP, just open up 100-1k TCP connections and multiplex them.
bish bash bosh, mostly line speed over a high latency line (mind you this was in the days of 100m-500m cross atlantic internet, you'll probably need more connections to saturate a 10gig line.)
Set larger kernel TCP send and receive buffers and enable BBR congestion control. Speed will usually be good also across high latency links, and no multiplexing logic needed. Especially if you control both sides of the connection.
I mean yeah, but that requires having access to the kernel config. so for most people multiplexing TCP is a useful way to maximise a link, without having to fiddle with stuff that is a pain to deploy. (politically as well as logistically)
I deployed this "technique" before BBR was a thing. It worked well enough for what I needed it to do (move large images from London to California) It was pretty simple to engineer as well (mainly because I didn't have to make a fancy custom error detection/correction/rate limiting system over UDP )
The hashgraph algorithm is pretty sweet too and doesn't have the issue of a single write leader like Paxos and Raft. Basically multi-writers / leaderless
https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf
But to be fair, I'm not certain that CAP theorem and partition tolerance really belong in a conversation about TCP anyway
Regarding ack not being received by sender when connection breaks, it's a weak and dishonest argument thinking it will strengthen their position, but completely ignoring the fact that TCP reliability is dependent on the simple and obvious fact that the connection exists!
I mean, that's true, insofar as pipes have incredibly weak guarantees too — after all, the other end of a pipe might be a program reading from/writing to a network socket, or other unreliable transport. Whenever you let your program be plugged into an arbitrary pipe, you have to expect all that same flakiness and then some.
Yeah pretty much.
maybe don't write contradictory unexplained nonsense.
Now that is a very interesting one!
It's sort of related to the question:
"How much of the Internet is accessible from any given point (location, locality, etc.) at any given point of time?"
Which is sort of unknowable, at least, without attempting to connect with every possible connection point on the Internet, which (if it could be done) would still consist of a range of time, and every point in time following that point would bring changes, perhaps small relative to the whole -- but accruing over time -- more and more, as more time elapses...
Observation: That same (or possibly similar!) phenomena would seem to be at play with respect to the measurment (observation) of quantum systems, i.e., the more certain you are of position, the less certain you are of velocity, and vice-versa...
Well, the more you measure the connectivity to all points of the Internet at one point in time, the less certain you might be of the state of the entire system as more time elapses from that point in time...
But now, why?
Observation: Generally speaking, the larger a system is, the more degrees of freedom it has, in attempting to "lock down" (know by observation, be "certain" of) the entire state of that system at one point of time, the more the parts of the system with degrees of freedom (how many degrees of freedom does the entire Internet have?) will change/evolve/move/"be subject to change" as more time evolves the state of the system... in other words, if you can know position (instantaneous state) with certainty then you can't know velocity (where it's heading to and/or future state and/or that which predicts future state) with certainty!
Sort of like you can know the instantaneous state of the Stock Market and its history... but no one can exactly predict its future (it has many, many degrees of freedom, all of which are subject to change in various unpredictable and bizarre ways!)
Which brings us back to #7:
>"7. If we fail to connect to a well-known remote host, then we must be offline."
We might be offline... but then again, we might not be! (Ping, ICMP, UDP, Telnet and Gopher anyone?)
But then again, we might be!
The Internet's online/offline status (is it really off if it is off? Is it really on if its on?) -- is much like some modern relationships, that is, "It's complicated!" :-)
The Internet is a Black Box!
It's Schrodinger's Internet!
You know, "if a TCP packet travelling at 99.44% of the speed of light on a westbound train track meets a UDP packet travelling at 99.43% of the speed of light on an eastbound train track, then when do they meet?"
You know, "solve for x..."
You know, "assume that the speed of light is constant and that quantum effects are not present!" :-)
A common problem are points which aren't really falsehoods, but where people frequently take false conclusions from it.
E.g. if you ask if TCP is reliable, especially in a non CS paper context, the answer is yes. That is iff you take a reasonable definition of reliable (which doesn't expect literally impossible things) and a reasonable interpretation of mostly. And just listing it as a falsehood fails to point out that there are two potential issues with your understanding while making creating the risk of someone with expertise in that sub-field of IT potentially thinking TCP is quite unreliable when it isn't. I mean the most common usage of the word reliable is a gradient with its meaning in a yes/no question being a short form of "reliable _enough_". Furthermore for most use-cases the "unreliable" aspect of TCP isn't even the main relevant misunderstanding people can have with "TCP is mostly reliable" (through for some use case it is)
The main troublesome misinterpretation is what mostly means. I.e. if you would give it a regious definition it would be "if sampling typical devices used in typical situations across some target audience then for most target audiences (weighted by audience relevance) most of the sampled devices will in a sufficient large long term moving average be reliable enough" or something like that.
What that mainly means:
- even if it's mostly reliable there will be devices for which it is reliable unreliable and anything in between
- similar even if it's mostly reliable for a device that isn't necessary all the time
- nor do we do statements about the patterns when the mostly doesn't apply, i.e. for a device TCP might be mostly reliable except every Sunday 3am for 30s, would still be mostly
- there are use-cases where unreliability is much more common
- there are audiences for which unreliability is much more common
etc.
Similar for point 5,6 about TCP packages, they are definitively a thing and there is no falsehood there. The falsehood is in believing you can reliable control them, that your OS or some middle ware isn't messing with them (e.g. splitting/combining/rewriting). So in some situations it's best to pretend there are non, but in some other situations you have to care and this might differ for different parts of the same protocol. So point 5 and 6 make sense, but don't point in a helpful direction.
to be clear doesn't mean lists are bad, or this list being particular bad, but I which they had more references/details even if short and compact and more clearly separate things too