Dup()'s shared file IO offset is a necessary part of Unix | Hacker News Reader

Dup()'s shared file IO offset is a necessary part of Unix

utcc.utoronto.ca

49 pointsingve10 comments

It's not clear if the author knows that there's a shared file structure in the kernel, and file descriptors simply point to this. When you open a new file you get a new file descriptor as well, so there's a 1-1 correspondence normally. But dup/dup2 exposes the indirection by only creating a new file descriptor that points to the old file (so this way you end up with multiple fds pointing to a single file structure). More interestingly this applies across processes, eg. after fork.

Here's the file struct in Linux (f_pos is the file offset):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

And here's the fd array in Linux (note file descriptors are indexes into fd_array which is an array of pointers to struct file):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

The author is talking about a cross-platform interface standardised by POSIX, it's not really the same thing as the details of one particular implementation. It isn't about how, but why

The gp isn’t only talking about a particular implementation. The concept and terminology of an “open file description” (represented by a struct file in Linux and BSD and others) in contrast to a “file descriptor” are standardized/documented in POSIX.

This is part of POSIX.

=> The dup() function provides an alternative interface to the service provided by fcntl() using the F_DUPFD command. The call dup(fildes) shall be equivalent to:

=> fcntl(fildes, F_DUPFD, 0);

-- https://pubs.opengroup.org/onlinepubs/9699919799/

Going to fcntl() docs,

=> The new file descriptor shall refer to the same open file description as the original file descriptor, and shall share any locks

-- https://pubs.opengroup.org/onlinepubs/9699919799/functions/f...

Which implies being exactly the same kernel data structure for the file description, with all its current state.

I love that the Linux kernel codebase is approachable enough that these things are easy to find

I was bitten by this recentishly and I had to learn the difference between file descriptors and descriptions. I had a program that was misbehaving when talking to a tty, but not when the output was piped through cat. Turns the program would set O_NONBLOCK (a description level property) on its input, but of course the shell was dup'ing the input tty fd to output as well, implicitly making the output non-blocking as well. Setting the output to a different description (i.e. the pipe) would workaround that.

For a long time `npm` would do that to your stdout/in, which didn't impact anything in interactive shells because your shell resets it every time it shows the prompt, but run those same commands in a bash script and later processes will crash with strange error messages because no one has ever tested with stdout being nonblocking. So much for separate processes being independent.

Well, the author of epoll(2) apparently didn't know that either, so...

I am certain that M. Siebenmann has known about file descriptions for decades.

The problem with a "here is what Linux does" response is that it actually bespeaks of the respondent not knowing enough. Linux isn't even relevant, as M. Siebenmann is talking of design decisions made 20 years before Linux was even conceived.

Whereas there were contemporary operating systems that used everything from file control blocks to file handles, and did not have these semantics. Nor indeed did they have the higher-level command processor semantics that they enabled.

The question M. Siebenmann was posing is in part whether it was inevitable that multi-process operating system designs ended up with these particular shared file pointer semantics, or just an accident of the fact that we got the idea of shared "standard" I/O streams long before we got the idea of forcing all sequential-access write operations to always append with a flag.

The issue isn't that this behavior is the default. The issue is that there isn't a portable way to also duplicate the file description. One has to open() the file multiple times or use platfom-specific things such as procfs.

Reply on news.ycombinator.com