Hacker News Reader

Comment by chuckadams | Hacker News Reader

It's fine, the content of an expanded variable isn't parsed further:

    $ touch "foo \"bar baz"; for f in *; do echo "$f"; done
    foo "bar baz

    # quotes don't affect it either
    $ touch "foo \"bar baz"; for f in *; do echo $f; done
    foo "bar baz

Though once you start passing args with quotes to other scripts, things get ugly. Rule of thumb is to always pass with "$@", and if that isn't enough to preserve quoting for whatever use case, write them out to a tempfile instead, or don't use a shell script for it in the first place.

View on HN

What about in the case of

  for f in `ls`; do echo "$f"; done

Same behavior, for the same reason?

The quotes are preserved, but backquote expansion fills the argument list using any whitespace as a delimiter.

    $ for f in `ls`; do echo "$f"; done
    foo
    "bar
    baz

If you absolutely must parse ls (let's assume it's some other script that outputs items with spaces) and the output can contain spaces, you have a few options:

    $ ls | while read f; do echo "$f"; done
    foo "bar baz

    # parens keep the IFS change isolated to a subshell
    $ (IFS="\n"; for f in `ls`; do echo "$f"; done)
    foo "bar baz

But if your filenames contain newlines, you'll really want to stick with the glob expansion, or output custom delimiters and set IFS to that.

> If you absolutely must parse ls

... stop and rethink your options. You may be able to get away with parsing the first columns of ls -l but even then a pathologically named file could make itself look like a line of ls output.

It's simply not possible in all cases. If you can constrain your input then you may be able to make use of it but in the general case, that's why xargs and find grew a -0 option.

Or glob.

Agreed when it comes to ls, but this applies to any script whose output you capture. I personally prefer “while read” loops but I’m probably screwed if someone smuggles in a newline.

If you are iterating over a lot of files, a read while loop can be a major bottleneck. As long as you use the null options from find and pipe into xargs, you should be safe with any filename.

I've found it can reduce minutes down to seconds for large operations.

If you have to process a large number of files, you can let xargs minimize the number of times a program is run, instead of running it once per file.

Something like:

  # Set the setgid bit for owner and group of all folders
  find . -type d -print0 | xargs -0 chmod g+s

  # Make the targets of symlinks immutable
  find . -type l -print 0 | xargs -0 readlink -z | xargs -0 chattr +i

Way faster. But there are lots of caveats. Make sure your programs support it. Maybe read the xargs man page.

Personally I skip the middleman when I can with "find ... -exec cmd {} +"

    find . -type d -exec chmod g+s {} +

Or even minimise arguments by including a test if the chmod is even needed:

    find . -type d \! -perm -g=s -exec chmod g+s {} +

I actually have a script that fixes up permissions, and I was delighted to fit it in a single find invocation which only performs a single stat() on each file in the traversal, and only executes chown/chmod at all for files that need change:

    # - ensure owner is root:shared
    # - ensure dirs have 775 permissions (must have 775, must not have 002)
    # - ensure files have 775 (if w+x), 664 (if w), 555 (if x) otherwise 444 permissions
    find LIST OF DIRS \
        '(' \! '(' -user root -group shared ')'              -print -exec chown -ch root:shared {} + ')' , \
        '(' -type d \! '(' -perm -775 \! -perm -002 ')'      -print -exec chmod -c 775 {} + ')' , \
        '(' -type f    -perm /222    -perm /111 \! -perm 775 -print -exec chmod -c 775 {} + ')' , \
        '(' -type f    -perm /222 \! -perm /111 \! -perm 664 -print -exec chmod -c 664 {} + ')' , \
        '(' -type f \! -perm /222    -perm /111 \! -perm 555 -print -exec chmod -c 555 {} + ')' , \
        '(' -type f \! -perm /222 \! -perm /111 \! -perm 444 -print -exec chmod -c 444 {} + ')'

But if you need multiple transformations of filenames in a pipeline like in your second example, then yes xargs will be involved.

`find` is almost always easier, but you can get quite far with `ls -Q` if you can assume GNU ls.

Thanks for that. For my reputation’s sake, let me clarify that I do always use either globbing or `find -print0` since a more experienced sysadmin drilled that into my head decades ago. I was curious about other edge cases, but I don’t take any convincing.

Reply on news.ycombinator.com