Hacker News Reader

Comment by fiddlerwoaroof | Hacker News Reader

I frequently find myself reaching for this pattern instead of xargs:

    do_something | ( while read -r v; do
    . . .
    done )

I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

View on HN

Some additional tips:

1. You don't need the parentheses.

2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope:

    i=0
    while read -r v; do
        ...
        i=$(( i + 1))
    done < <(do_something)

The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯

[1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

Yeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too.

I think the redirection can come first, though (not at a computer to test):

    < <( do_something ) while read . . .

Yeah, for commands, the input/output redirections can precede them, but for some reason it doesn't work for builtin constructs like `while`:

    $ < <( echo foo ) while read -r f; do echo "$f"; done
    -bash: syntax error near unexpected token `do'
    $ < <( echo foo ) xargs echo
    foo

    $ bash --version
    GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin20.2.0)

Maybe wrap the loop either with parentheses or braces?

Tried that, but nope :D I'll let you figure this one out once you get near a computer!

Redirection like this doesn't seem to work if it comes first on GNU bash 5.0.17(1)-release.

For documentation purposes, this is the exact thing I tried to run:

    $ < <(echo hi) while read a; do echo "got $a"; done
    -bash: syntax error near unexpected token `do'

    $ while read a; do echo "got $a"; done < <(echo hi)
    got hi

Maybe there is another way...

One way which isn't great, but an option nonetheless… The zsh parser is happy with that form:

    $ zsh -c '< <(echo hi) while read a; do echo "got $a"; done'
    got hi

My position isn't that it is a good reason to switch shells, but if you're using it anyway then it is an option.

I’ve always preferred zsh and, as I’ve slowly adopted nix, I’ve slowly stopped writing bash in favor of zsh

This is not POSIX compliant though.

These days bash and/or zsh are available nearly every place I care about, so I find POSIX compliance to be much less relevant.

No, process substitution must be provided by the kernel/syslibs, it is not feature of bash. For example there is bash on AIX, but process substitution is not possible because the OS do not support it.

ksh93 depends exclusively on the kernel implementation of /dev/fd devices. I just checked `cat <(ls)` a moment ago on both Linux and AIX 7.2--the latter fails in ksh93t+.

Bash uses /dev/fd when available, but also appears to have an internal implementation which silently creates named pipes and cleans them up. In Bash 5.0.18 on AIX, fake process substitution works just fine, in my testing.

Yes, you are right. Bash 5 on AIX 7.2 works with process substitution. Thanks for the advise!

I use this exact pattern a lot. One thing to consider is that in the process substitution version, do_something can't modify the enclosing variables. The vast majority of the time I want to modify variables in the loop body and not the generating process, but it's worth keeping in mind.

One common pattern I use this for is running a bunch of checks/tests, e.g.

    EXIT_CODE=0
    while read -r F
    do
        do_check "$F" || EXIT_CODE=1
    done < <(find ./tests -type f)
    exit "$EXIT_CODE"

This is a more complicated alternative to the following:

    find ./tests -type f | while read -r F
    do
      do_check "$F" || exit 1
    done

The simpler version will abort on the first error, whilst the first version will always run all of the checks (exiting with an error afterwards, if any of them failed)

I usually write zsh scripts and I think there’s a shell option in zsh that allows the loop at the end of the pipe to modify variables in the enclosing body: I remember at least one occasion where I was surprised about this discrepancy between shells.

Interesting! Indeed, Greg's BashFAQ notes it too: https://mywiki.wooledge.org/BashFAQ/024

>Different shells exhibit different behaviors in this situation:

>- BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').

>- BASH, Yash and PDKsh-derived shells create a new process only if the loop is part of a pipeline.

>- KornShell and Zsh creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. The read example above actually works in ksh88, ksh93, zsh! (but not MKsh or other PDKsh-derived shells)

>- POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).

Check out bash lastpipe option.

Also for the `while` enthusiasts, here's how you zip the output of two processes in bash:

    paste -d \\n <(do_something1) <(do_something2) | while read -r var1 && read -r var2; do
        ... # var1 comes from do_something1, var2 comes from do_something2
    done

For thousands of arguments this sloution is much slower (high CPU usage) than xargs, because either it implements the logic as a shell script (slow) or it runs an external program for each argument (slow).

Sure, if performance matters use xargs. I find this is easier to read and think about.

Thank you. Your comment coalesced a number of things in my mind that I hadn’t grasped properly as a UNIX midwit, especially the braces thing.

creating a subshell can lead to some surprising behavior if you aren't careful though.

Reply on news.ycombinator.com