Two File Descriptors for Sockets
cr.yp.to
cr.yp.to
Then again, in most of my "non-trivial" network code I already end up cracking the socket into two pieces anyhow, and while I take advantage of the pun that closing either of them closes the other it would be trivial for me to add this behavior to the separated sockets if that's what I wanted. Partially this is a result of my "non-trivial" network code being in Go so I end up running a goroutine each for send and receive, but still I imagine this would be a common structure one way or another in a lot of non-trivial network servers.
But the additional states on the TCP socket would definitely bite some people. Even in its current simpler state it still confuses people and there's a post about how some server turned out to be leaking sockets because it didn't properly close them shows up with some frequency on HN.
int sock_fd, split_pair[2];
char c;
int n = socket_split(sock_fd, split_pair);
close(sock_fd);
n = write(split_pair[1], "a", 1);
close(split_pair[1]);
n = read(split_pair[0], &c, 1);
read(split_pair[0], &c, 1); /* returns 0 / EOF */
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
If the splitting is optional, as it would have to be now, and as I proposed it would not add complexity since it would be opt in.
Theoretically if it had split from the start, it may have added a minor amount of complexity to some programs, but not a significant amount. I don't get the need to add TCP states, as the TCP machine will not change, and the user space app shouldn't really be tracking/replicatting the TCP states.
Anyway, your assumed (and mandatory) point of view was not obvious to me, I interpreted it from the position of "if that was done now". Which would add a trivial amount of logic to the kernel, mainly reusing existing logic, structures, and counters.
So I guess we were slightly at cross purposes.
Huh? Half-closed connections already exist. The complexity in the state machine is there, and more TCP users should be aware of it (the source of soo.... many... bugs)
This wouldn't necessarily manifest as a new TCP state in the kernel state table. You could do this to TCP today at the API level alone. Which is further evidence it's a different "half".
And if it sounds silly that anyone would make such a mistake, like many such hypothetical mistakes it's important to remember that in the wild it's not always next to each other. My least favorite networking experience ever involved being three levels deep in abstraction in some Perl libraries, every level of which felt like it can and should make all sorts of helpful little calls, the end result being all kinds of fun to debug as soon as any of them were wrong. The thing that closed one of the halves of the socket and the person confused by the result can be arbitrarily separated in the code.
Or, to put it another way, you can't grow the API's complexity without growing the API's complexity. If today closing closes both reading and writing but a new API severs that guarantee and creates at least two new states, that inevitably results in a new way for things to go wrong. You can't have one without the other.
shutdown(sockfd, 1)
gives you "I'm able to read from my socket but not write from it, what gives?"
The real bugs with TCP come from "why am I not able to download the file because the server did close(sockfd) rather than shutdown(sockfd, 1), and my TCP stack ACK'd, causing the server to RST, causing my machine to give me an error before I read all of the data".
This is already waaaay harder than the counterfactual you're imagining, and the bugs are here, today.
The problem in the presented example is that there are, fundamentally, 3 communicating entitys A->B->C, which logically demands two channels. A quirk of the example where A and C happen to be on the same system and coincidentally can use the channel disjointly (read-only and write-only) in conjunction with TCP being inherently two-way allows it to be kludged into a single channel, but frankly it is your fault for relying on bizarre implementation details [1].
If you want a proper abstraction, what you would actually do is multiplex multiple logical streams over a single duplex channel. The code would then operate purely on the logical streams (pipe-like) and everything would be clean. The implementation would then only need to establish connection once, removing the inherent overhead associated with establishing multiple network protocol connections, while still giving you a clean and consistent abstraction (barring the standard leakiness of the TCP abstraction).
To clarify, would one be send and the other be receive
or
are you saying one would be for the application data and the 2nd would be for the protocol SYN/ACK and all that?
I've created a few communication protocols in my time (all people of a certain age have, because we pre-date TCP/IP), and I am reasonable familiar with TCP itself. I don't see the extra complexity you speak of. At most, it is de-multiplexing the ACK fields in the header from the rest. If you want to whinge about unnecessary complexity in the TCP protocol, I'd suggest starting with it being byte orientated rather than packet orientated.
Secondly Dan is out of touch. He's right in saying it's a complex solution for a shell pipeline, but the TCP is designed to operate in a less reliable world. In TCP's world you can no longer assume a message you sent was acted upon by the other end. The network could have died, or the system crashed before the data made it to disk. Thus the shell command:
echo Run for your life or you will die | mail friend@doomed.com
It looks like TCP's successor will be QUIC. In QUIC there isn't just two streams, you can create lots of sub-streams, using datagrams if you want. Now that is a lot of extra complexity. But from the applications point of view it's a very nice feature. It's a bit like the second channel TCP gives you, as it's surprising how often you need a more than one channel between two programs. In fact the very first use TCP was put to was FTP and guess what - FTP creates two TCP connections. And it's example of how painful it can be to set that up as NAT broke the original FTP for a while. We now finally have an off the shelf solution in QUIC. It only took 30+ years to arrive.
It looks to me like the world has voted. The outcome is a firm Nay to Dan's suggestion, and it's now moving in the opposite direction.
NAT broke FTP for about the 5 seconds it took for one to realise they needed to type in the PASV command - which existed right from the start.
It was in RFC 542 (1973) - pre TCP, and RFC 765 (1980) - first post TCP.
NAT dates from around 1994.
Re the reliability: the same is true of pipes. You can't assume `mail` will take the message, even with pipes, because, just as with TCP, there's no feedback channel. The disk could have filled up after `mail` took the final bytes from its input, for example.
Your point about application-level acknowledgements is 100% spot on, then, but pipes need application-level acks too, in much the same circumstances as TCP would, for just the same reasons. Even pipes make a distributed system.
Not all protocols need application-level acks, though. In the `echo | mail` example, you do get feedback that the process exited abnormally, but this is feedback to the supervisor (here, the shell), not to the preceding process in the pipeline. The same could (and should) be arranged for a TCP-based link between `echo` and `mail`. The failure-signalling takes on some of the responsibilities than an app-level ack would have discharged. Pipes vs sockets makes no difference.
Your point about the difficulty of constructing the necessary back-channels for app-level acks is also spot on. If one were to reformulate a transport layer in terms of simplex channels, one would want to retain an easy way to configure a pair of them as a single duplex channel. This is as true of pipes as of sockets. A nice design would allow capability-passing as well as regular message sending: then, you'd open an outbound stream to a server, and as your first message, you'd send a capability for the write-end of a matching inbound stream. Modular session establishment. Permits a bunch of other interaction patterns, too, such as session handoff, redirection, continuation-passing, load-balancing etc. Richer topological possibilities even than SCTP and QUIC. If capability-passing was in TCP, it'd have been trivially handled by the NAT, too, so FTP wouldn't have been broken. Ah well, never mind.
All that said, Dan isn't taking aim at TCP-the-protocol here. (He does, elsewhere.) Instead, the focus is the sockets API. The API could easily have involved two unidirectional fds instead of just one, without changing the protocol, or how easy/difficult it is to establish a connection. It'd just expose a read cap and a write cap instead of a single mixed capability. It'd be a small change that'd eliminate a bunch of special-casing. I think he has a good point; it's a shame we can't get there from here...
I can't see how it's an improvement. The pipe code is invariably:
pipe(fds);
close(fd[0]); /* Close the one we don't need. */
connect(soc, ...);
shutdown(soc, ...); /* Close the one we don't need. */
This makes me think of hobgoblins and minds.
So there's a downside to the inconsistency. Is there a corresponding upside I'm not seeing? Cause if not, perhaps all else equal, in which case I'd rather have the static simplification.
True. But code like this is invariably a 3 step process - setup the invariants, loop (generate-data), then close down. The steps above happen in the open, which means `generate-data` doesn't need to know what to call.
The Dan's pipeline gave is a good example of this as any. It is the shell that does the open step, so it's the thing that does the open, so it creates the pipes and does the close() of the unwanted end. Thus Dan `generate-data` does not need to know whether it is sending over s pipe or TCP connection.
In fact bash accepts /dev/tcp/host/port syntax. I presume it already does the required shutdown as part of that of setting it up. Which demonstrates that in real life `generate-data` in indeed blissfully unaware of whether it's a pipe or socket.
Finally, notice the open step is always going to be different, so this additional work of setting it up different is not a big imposition. It's has to be different because it has to connect the two ends, and that is going to be different depending on whether they are local or remote.
This is not a pipe vs socket issue only but a penalty for having a vfs abstraction. Files, pipes, sockets, and more all have stream like behavior but they can vary widely and many cases demand leaking the vfs abstraction to check what's the fd type.
In the first example, he is using two pipes; in the second example, he is using one socket. And he complains that that they don't work the same!?!
The problem is that they saw TCP sockets are two-way and then thought to optimize the TCP connection/state overhead by using it for double-duty without applying appropriate abstractions to transparently use a two-way channel as two one-way channels.
An early FTP protocol (452, 454, 542) originally used a server listen port of 3 (and hence a server "return port" of 2*), but 542 allowed temporary use of port 21 as the protocol was changed. A different pair of ports (even or odd depending upon TX or RX end) were used for the data channel, and the port picked was related to the Telnet protocol port in use.
RFC 640 then seems to have fixed the use of port 21 instead of 3. Then when TCP came in to use RFC 765 seems to have simplified things somewhat, sticking to port 21.
Also see RFC 165 for the/a Telnet Initial Connection Protocol showing the use of the two even/odd ports for forming a bidirectional connection.
* Or 4, but I believe the return channel was one port below, not above.
Even being receive, odd transmit. That then led to various higher level protocols using pairs of ports. Once TCP took over, that was relaxed.
The best mitigation is to not close `stdin` when you get EOF on it -- just stop reading from `stdin`.
https://cr.yp.to/ucspi-tcp.html
These are great for duct-taping connections. I created a whole suite of utilities for another programs for uploading firmware updates. I was able to create one program for handling the uploads with several different comms protocols and devices, without having to handle details of tcp connections, serial connections, UDP, etc. socat is another great tool for this.
The insight that a TCP connection is basically equivalent to a pipe is a pretty good demonstration of how this could have been better.
As another related example, imagine if the same had been done for standard I/O. What if instead of having separate file descriptors, all of the standard I/O happened through one file descriptor? Imagine how much more painful everything would be.
Unfortunately, this is now pretty much how a lot of stuff works, but it did not really have to go this way.
Also, it's a general interface. It's easy to imagine exotic protocols where shutdown() is not so cleanly equivalent to close() as it is for TCP.
TBH I find it odd to focus on a preference for one semantic over the other. The real thing we can all probably agree on is that in a perfect world, they ought to be the same.
If he would have done that it would have closed the socket only once both fd were closed and the fds would have separate lifetimes like in the pipe example.
duping the fd's as you describe would achieve nothing, as both fds will share the same file, and hence while either fd is open, the kernel can not detect a close.
The process-data task is on a remote machine, probably listening on a single port, and feeding its output back over the same TCP connection as its input.
Some process has then set up the series of connections, by connecting to the remote server, plumbing and doing fork/exec to run consume-data with the socket on stdin, while doing an exec of generate-data having the socket on stdout.
Now the processes being run are assumed to be general unix processes, i.e. pipeline processes ala how cat, head, grep etc operate. They generally read from input, write to output, and exit when the input is exhausted. That "exit signal" will then be fed through the pipeline as a series of "EOF" notifications, which FIN can achieve for TCP.
However because the output fd from generate-data, and the input fd to consume-data both share the same underlying socket, the mere close of the output fd from generate-data exiting will not result in a FIN being sent to process-data.
That is because the same "file" (the socket) is open by consume-data (in read/write mode), and so the kernel can not know that consume-data will not write to the socket.
If the UNIX abstraction for the sockets had the ability to split the socket across to "files", not just fds, then the issue would be removed; but it doesn't.
Which is what the author wants. He wants the socket to close once both generate-data and consume-data are done with the socket.
However for that to work, the two FDs (generate-date to process-data, and process-data to consume-data) must not be associated with the same "open file", even though they could (with a change) represent the same TCP socket.
i.e. he wants generate-data to exit, closing its output FD, and have that drive EOF all the way through to consume-data. Such that consume-data does not determine it is done until it sees EOF on its input socket.
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
Plan 9 with its protocol 9p is the wired version of inter-machines communication. You don't just link two random processes, you have the possibility to interconnect two directory trees, and since most useful software works through a directory tree, this means two processes can be linked by importing another tree on another machine and doing everything locally. It's possible to write something like "echo UNSEEN | imap-search | imap-get-summary" and have actual IMAP stuff done on another machine, and you don't see any of it.
It is also possible to use `rcpu` to do what ssh does, and even more: the remote machine has access to the local machine. So for instance you can edit files locally, and have a rcpu connection to run builds on another machine that picks it from your local workstation. No clunky sshfs, no failing NFS, ...
Also the tty is typically connected to stdin, stdout and stderr, that are separate file descriptor (again, a leaky abstraction when they refer to the same description).
...can curl already do that?