Comment by mmastrac | Hacker News Reader

Comment by mmastrac | Hacker News Reader

There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related.

https://forums.unraid.net/bug-reports/prereleases/sqlite-dat...

From that report:

====== 8< ======

> In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD which tells I/O driver this is a read-ahead. In this case, if there are insufficient resources at the time this request is received, the driver is permitted to terminate the operation with BLK_STS_IOERR status. Here is an example in Linux md/raid5 driver.

> In case of Unraid it can definitely happen under heavy load that a read-ahead comes along and there are no 'stripe buffers' immediately available. In this case, instead of making calling process wait, it terminated the I/O. This has worked this way for years.

[...]

> What I suspect is that this is a bug in SQLite - I think SQLite is using direct-I/O (bypassing page cache) and issuing it's own read-aheads and their logic to handle failing read-ahead is broken. But I did not follow that rabbit hole - too many other problems to work on :/

====== 8< ======

Some related bugs they brought up:

https://bugzilla.kernel.org/show_bug.cgi?id=201685

https://patchwork.kernel.org/patch/10712695/

Edit: it appears that there's _something_ causing corruption when drivers fail read-ahead I/O. Whether it's SQLite or something else in Linux is another question.

> I think SQLite is using direct-I/O (bypassing page cache)

Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no?

    strace -e open -o >(grep O_DIRECT) ./myprogram

  -o >(grep O_DIRECT)

This is cool. Never seen it (but it makes sense.)

Its cousin

  diff <(cmd a) <(cmd b)

is also very handy.

Not a command line guru, but this looks interesting. Does it compare outputs of two commands? How does it do it?

<(command) is being replaced by a path to a file which contains the output of the command:

  $ ls <(echo A)
  /dev/fd/63

  $ cat <(echo A)
  A

So

  diff <(command1) <(command2)

becomes

  diff /dev/fd/62  /dev/fd/63

It is called process substitution and specific to Bash (not POSIX shell compliant). The output of the commands in parenthesis will become runtime (file) inputs for the diff command.

More: https://www.gnu.org/software/bash/manual/html_node/Process-S...

It's not specific to bash -- it's in ksh as well. And zsh.

I'm also pretty sure it originated in ksh, not bash.

diff compares files line by line, here we just redirect the output of cmd1 and cmd2 via <, so they serve as 'proxies' for the file-args diff expects

you might want to read into redirection on the commandline, it's amazingly powerful

It might be easiest to view it like so:

  $ echo <(echo 1) <(echo 2)
  /dev/fd/63 /dev/fd/62

The shell spawns two commands connected to pipes, then replaces those with a file that represents the other (read) side of that pipe.

The command above with >(grep something) does the same thing, just with the other end of a pipe.

It's not actually a pipe. >, < and | are IO redirection commands; but of those only | is a pipe. Pipes specifically "pipe" the STDOUT of one command as the STDIN of another. The other two instead point a command to the file descriptor of the output of another.

This is explained in more detail here:

https://askubuntu.com/questions/172982/what-is-the-differenc...

On linux at least, these two constructions are effectively identical from the view of the "piped" process.

  # cat <(ls -l /proc/self/fd/)
  total 0
  lrwx------ 1 root root 64 Jan 12 15:02 0 -> /dev/pts/0
  l-wx------ 1 root root 64 Jan 12 15:02 1 -> pipe:[20519634]
  lrwx------ 1 root root 64 Jan 12 15:02 2 -> /dev/pts/0
  lr-x------ 1 root root 64 Jan 12 15:02 3 -> /proc/23611/fd/

  # ls -l /proc/self/fd/ | cat -
  total 0
  lrwx------ 1 root root 64 Jan 12 15:02 0 -> /dev/pts/0
  l-wx------ 1 root root 64 Jan 12 15:02 1 -> pipe:[20518265]
  lrwx------ 1 root root 64 Jan 12 15:02 2 -> /dev/pts/0
  lr-x------ 1 root root 64 Jan 12 15:02 3 -> /proc/23621/fd/

Yes, because “cat” can use either STDIN or a file as input:

https://github.com/coreutils/coreutils/blob/master/src/cat.c...

It’s an intentional design decision.

I think we're probably in agreement, but we're getting caught up on the definition of "piping".

In my case I'm using it to describe the use of a kernel pipe object. In your case, you are using it to describe the higher-level concept of connection from one processes stdout to another process's stdin (or something along those lines).

I think the point is that the directory listing is the same, not that cat supports both syntaxes.

Repeating shawnz's comment more explicitly: you said that the mechanism used by <() is "not actually a pipe". What is `pipe:[20519634]` doing in the output of `cat <(ls -l /proc/self/fd/)`?

What it looks like is ls having its stdout (fd 1) directed to a pipe.

It is a pipe, just for a different definition of pipe.

| is the pipe operator in sh for doing a shell "pipeline". However, the parent comment is referring to a linux pipe as in pipe(7) [0].

One easy way to see this is with the following:

    $ ls -l <(echo foo)
    lr-x------ 1 user group Jan 12 01:23 /proc/self/fd/16 -> 'pipe:[2937585]'

As you can see, that command created a fd (16) which referred to a pipe (pipe:[2937585]).

Those file descriptors were created using the pipe(2)[1] call by the shell, so it seems fine to refer to them to pipes.

I'll also note that <() / >() are _not_ using the "redirection operator". They're actually distinct operators for "process substitution"[2] in bash terminology. They look similar to redirects, but they're not the same operator, so that stack overflow answer isn't really relevant.

[0]: http://man7.org/linux/man-pages/man7/pipe.7.html

[1]: http://man7.org/linux/man-pages/man2/pipe.2.html

[2]: https://tldp.org/LDP/abs/html/process-sub.html

On POSIX systems, the line between file and pipe doesn’t exist. If you create a named pipe w/ mknod or mkfifo, you would interact with the named pipe as you would a normal file. Or block/character device. Etc.

In case of unnamed pipe the fds just exist as long as the process is running, then it disappears. Point is it’s irrelevant what you refer to as pipe, at the end of the day some file-like object is either written to or read from. And really, that file-like object is actually just a buffer.

The line absolutely exists, it's just behind an abstraction barrier for read(), write(), and close() calls. The creator of a pipe calls pipe() or pipe2(), and holders of the fd can get access to pipe-specific features with fcntl() and ioctl().

Does anybody know if this can be done in Windows (not WSL)? E.g. is it possible to open a named pipe (?) through its name, and work with it as a sequential file?

This isn’t a Linux thing, it’s a bash thing. The <() syntax isn’t using a named pipe — it is creating an anonymous pipe. But you can replicate the same functions with named pipes.

A quick search says that Windows has support for named pipes as an IPC mechanism [1], but I’ve never used them. But this is a Windows API. I am not aware for any way to do this with plain cmd.exe. You’re not the first to ask though... [2]

[1] https://docs.microsoft.com/en-us/windows/win32/ipc/named-pip...

[2] https://superuser.com/questions/430466/in-windows-can-i-redi...

Are you sure? How does diff know to read from those two pipes?

Even the echo example hints, that the created pipe has a name, and that name is then simply passed as the argument to the command.

> Even the echo example hints, that the created pipe has a name

Well, the echo example shows that the pipe is reified at /dev/fd/63 (or /dev/fd/62 for the other pipe).

That's not a named pipe. It's a file descriptor identified by the integer that defines it, accessed under the /dev/fd listing of all file descriptors.

So, to sort of answer your questions:

> How does diff know to read from those two pipes?

They're two different pipes. Diff takes two arguments. Each argument is one of the pipes.

> (implied) Is there such a thing as an anonymous pipe, in any context?

Not if you consider /dev/fd/id-of-pipe to be a name. The operating system lists them all there. But usually you would use a named pipe if you wanted to be able to know the name, so that you could find it again if you didn't already have a reference to it. If you jammed your finger into my chest, I would be there, and my physical form would block the passage of your finger analogously to how an anonymous pipe nevertheless exists as an entry (two entries?) in /dev/fd. But that wouldn't tell you my name in the conventional sense, even though it would be a valid and unambiguous way to refer to me.

The <(foo | bar) syntax evaluates to a file path, e.g.

  $ echo <(echo hi)
  /dev/fd/63

If a program reads from that file path, it gets the output of the command chain.

  $ cat <( (echo foo; echo bar) | grep bar )
  bar

Of course, that's a silly example. But that explains how the diff example works, since it's the same idea: diff just reads from the two "files".

EDIT: Heh, 5 different replies within the same 60 second window.

Yes. You're directing the STDOUT of each command as a FILE into diff. Diff takes two files, thus the duplication.

it does[0] you are redirecting stdout of both of the subprocesses (the things in brackets) to the stdin of the diff command (I think it creates two different input streams, that diff knows how to compare).

[0] http://tldp.org/LDP/abs/html/process-sub.html

It also has a great unofficial name: the penguin operator.

<()

See? Penguin!

I don't think sqlite uses direct IO -- at least there's no mention of O_DIRECT in the source code

SQLite doesn't use direct I/O.

Reply on news.ycombinator.com