TCP is UNreliable | Hacker News Reader

TCP is UNreliable

cliffc.org

26 pointsLiveTheDream19 comments

I voted this up (tried to submit, actually), but is this article correct, or is something in this person's stack not actually programming sockets correctly?

When you call shutdown() to arrange a FIN for a socket, especially after you've sent data and received no application-layer acknowledgement, you're supposed to verify that the connection closed properly. You do this by select()ing on the socket and then read()ing 0 bytes.

The error condition (the RST segment they're getting) is signaled by the "Connection closed by peer" errno --- and -1 from read().

The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.

> The reliability signals this app needs are built into both TCP and the socket programming interface, from what I can tell. The application-layer acknowledgement it wants appears duplicative.

An ACK only suggests that a segment has reached the recieve buffer successfully. If the receiving application crashes between buffering incoming data and successfully processing it, that's expressly not TCP's problem. The RST will signal that the receiver has died, but does not imply whether or not the application has successfully processed any buffered data up to that point.

AFAIK, an application-layer acknowledgement is the only way to solve this problem.

I agree that there are circumstances where you need application-layer acks, SMTP being a canonical example.

However, there are other cases where you don't need lock-step app-layer acknowledgements; for instance, if your messages are idempotent.

I'm really just reacting to the forensic work in this article, which suggests (maybe I'm misreading) a problem that TCP doesn't actually have.

Yes, if all you want is to confirm that a TCP connection has closed and flushed correctly, then AFAIK, that's entirely possible with a [shutdown, select, read] sequence.

I struggle to understand why an RPC system (as detailed in the article) wouldn't want an app-level acknowledgement though...

Without looking at the code, it's hard to say, typically errors like the one described won't be caught by return values of read/write, but by return values of close() -- which many many people don't bother to check. (Not saying that's what happened, but that's my bet)

Did they close() the socket? If you write data to a socket, and receive no signal that it's been processed, shouldn't you shutdown(2), select(2), then read(2) before you close(2) it?

> after you've sent data and received no application-layer acknowledgement

This would be the most robust fix. Have the server return "OK" or something after it has processed the data. It could have silently crashed but still accept connections.

I'm not sure I see how this is more robust than simply checking to make sure you had an orderly close. The orderly close check should also catch the case where the application crashes after its TCP stack has received the data but before the application processed it.

I'm not saying there's never any point to application acknowledgements; look at any careful SMTP server implementation for a study of how to handle these problems. TCP can promise a reliable transport but obviously not a reliable disk.

Though, the contract you have with TCP , is between TCP at your end and the TCP at the peer, not that it also have delivered the data to the application of the peer.

If the peer crashes just as it read() the data, but didn't process it, it will look just as a normal clean close on the sender machine if there arn't more data laying around in the TCP buffers.

To be clear, you're talking about the case where the application dequeues the data from its receive queue and then crashes. If you close a socket without reading queued data, you get an RST.

This doesn't sound like a TCP problem.

Mistake #1: The sender optimistically sends a first packet before getting a SYN reply. That implies that the sender issued a non-blocking connect, and then wrote without waiting for the connect to complete. Otherwise, the sender (kernel) wouldn't have any data to send right away.

Mistake #2: The sender sends a FIN, but doesn't block because all data has already been acknowledged. This is supposedly a problem because the app never receives the data, but a TCP ack has never meant that data got beyond the receiving kernel to an app.

So yes, it's true that neither side ever got an error and unfortunate that it's possible for the receiving app to get no data in that case, but that doesn't make it a TCP problem. Somebody at a higher level - JVM or the app itself - over-optimized their "connection and first data" code without understanding the actual behavior of the TCP protocol and socket API. Then they "fixed" it by working with their own protocol instead of against the standard one.

Nothing to see here. It was just a false alarm. Move on.

I think you're right, but TFA appears inconsistent: it says "no errors of any sort...All standard blocking calls." As you point out, it later says that the failed case optimistically sends data after the first SYN, before the SYN/ACK back, but that's not possible if connect() is being used in blocking mode. (Right?)

It's not a TCP problem. It's a problem with the linux implementation of TCP.

No, it's not even that. The linked article (http://www.cliffc.org/blog/2013/08/15/tcp-is-unreliable/) at the bottom tells you what's actually going on.

The only thing slightly fishy is that the peer application didn't receive any connection attempt, but if you're anyway quite overloaded, e.g. running out of file descriptors, that may well happen.

The FIN/SO_LINGER problem is real, and often misunderstood, but I don't think that was quite the whole story here. SO_LINGER would flag the error some of the time, but not always (see Mistake #2). Even if it did, avoiding the problem altogether (see Mistake #1) would be better than flagging it after the fact. Even with SO_LINGER on and the kernel's RST issues fixed, the app would still be erroneous.

I still don't get how this can happen as described. If the client is using a blocking connect and checking for errors (as the author says it is), I don't see how that can succeed without the remote side at least seeing the connection. As you say, the receiver could be overloaded and ignore the connection, but the client should get a connect failure in that case.

The remote kernel sees the connection fine. It accepts it, receives the data, the client closes its end. What might not happen is, in a very resource strained condition, is for the remote application to get the data. There's no relation between you doing a connect() and the remote doing an accept() in TCP, it's handled asynchronously by the kernel. (Traditionally the backlog argument to listen() controlled this behavior, but that backlog argument is by and large ignored on modern operating systems)

The caveat here is that you close() the socket when you're done. close() does not, by default, wait until the remote end also closed its end. In the scenario described above, the remote end would eventually signal an error, but if you shoved the data into the remote kernel buffers, close and forget about that connection, your app will never see that error.

Note that he links to http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-s... which more or less explains what's going on.

Interesting! One thing that I don't understand (which might be a little off-topic): If it's an RPC call, shouldn't there be a response for each RPC request to ack that the RPC request is well processed by the application layer? Otherwise how do you make sure the application layer won't screw up? How do you notify the client that the request is not invalid?

Reply on news.ycombinator.com