Hacker News Reader

Comment by f311a | Hacker News Reader

Most people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility.

When using jq, I can do a lot of things:

    aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort

View on HN

Another common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go.

That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)

Which is sorta-kinda the same thing as

  jq .field <(aws s3 cp s3://bucket/file.json.gz - | zcat | head) | sort

which is more annoying to type but works.

Redirection is not a parameter, meaning that would still not work with this Q tool.

This is not a simple redirection. cmd <(subcmd) is a bashism that redirects the output of command subcmd to something that looks like a regular file to command cmd. Command cmd receive a path at the place the <(subcmd) syntax is used. Different from cmd < f, which redirects the contents of file f to cmd's input.

So, this should work :-)

Works in other shells, not just bash

  % cat /proc/self/cmdline <(echo $SHELL) | tr '\0' ' '
  cat /proc/self/cmdline /proc/self/fd/11 /bin/zsh

Yep! Zsh supports a lot of bashisms :-)

It won't work in dash though, and you should not use this in a shell that targets POSIX.

Reply on news.ycombinator.com