Hacker News Reader

Comment by probably_wrong | Hacker News Reader

I think there's a middle point where you want to do something that's complex enough that a glob won't cut it but simple enough that switching languages is not worth it.

I think the example of "exclude these two types of files" is a good case. I often have to write stuff like `ls P* | grep -Ev "wav|draft"` which doesn't solve a problem I don't have (such as filenames with newlines in them) but does solve the one I do (keeping a subset of files that would be tricky to glob properly).

In my experience 95% of those scripts are going to be discarded in a week, and bringing Python into it means I need to deal with `os.path` and `subprocess.run`. My rule of thumb: if it's not going to be version controlled then Bash is fine.

View on HN

You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"`

This uses regex to match files ending in .wav or .draft (which is what I interpreted you to want). Xargs then processes the file. You could use flags to have xargs pass the file names in a specific place in the command, which can even be a one liner shell call or some script.

So the "find <regex> - xarg <command>" pattern is almost fully generally applicable to any problem where you want to execute a oneliner on a number of files with regular names. (I think gnu find has no extended regex, which is just as well- thats not a "regular expression" at that point)

> You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"`

Find can even execute commands itself without using `xargs`:

     find -maxdepth 1 -iregex '.\.\(wav\|draft\)' -exec echo "found file:" {} \;

Definitely do it this way if you want to stick to the pre-filtered version (I recommend the cousin comment, filter inside the loop). GP's version is buggy in the same way as the post misunderstands, particularly with files that somehow got newlines in the filename (xargs is newline-delimited by default).

If for some reason you do need the "find | xargs" combo (maybe for concurrency), you can get it to work with "find -print0" and "xargs -0". Nulls can't be in filenames so a null-delimited list should work.

As an addendum, note that `-print0` and `-0` for find and xargs respectively are now in the latest POSIX standard, so their use is compliant.

The latest standard I know of is SuS 2018, which I have the docs for, and does not include either switch. I searched around a bit and it doesn't seem like there is a new one. Are you referring to some draft? I sure wish this was true.

That being said, I would interpret "-exec printf '%s\0' {} +" as being a posix compliant way for find to output null delimited files. I say this since the docs for the octal escape for printf allows zero digits. However, most posix tools operate on "text" input "files", which are defined as not having null characters. Thus I don't think outputting nulls could be easily used in a posix complaint way. In practice, I would expect many posix implementations to also not handle nulls well because C uses null to mean end of string, so lots of C library calls for dealing with strings will not correctly deal with null characters.

The 2024 standard is out, but behind a paywall; I don't know when they will update the Open Group website.

Geez. What's the point of writing standards if they won't let people read them (without having to pay for the privilege)?

>xargs is newline-delimited by default

Even worse, it is whitespace delimited (with its own rules for escaping with quotes and backslashes)

It is not, but (for reasons unknown to me) it doesn't quote parameters in the default mode. Consider:

touch "a b" ls | xargs rm # this won't work, rm gets two parameters ls | xargs -i,, rm ,, # this will work

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/x...

>[..] arguments in the standard input are separated by unquoted <blank> characters [..]

As for -i, it is documented to be the same as -I, which, among other things, makes it so that "unquoted blanks do not terminate input items; instead the separator is the newline character."

Yeah, I misremembered. Here's an example, using "-n 1" so each split "thing" is passed to separate processes:

  $ printf "one two three\nfour five\n" | xargs -n 1 echo
  one
  two
  three
  four
  five

>GP's version is buggy in the same way as the post misunderstands, particularly with files that somehow got newlines in the filename

I understand this caveat, but I never had a file with newline that I cared about. Everyone keeps repeating this gotcha but I literally don't care. When I do "ls | grep [.]png\$ | xargs -i,, rm ,," (yes, stupid example) there is 0% chance that a png file with a newline in the name found itself in my Downloads folder. Or my project's source code. Or my photo library. It just won't happen, and the bash oneliner only needs to run once. In my 20 years of using xargs I didn't have to use -0 even once.

See the other response I got, I misremembered (and waaay too late to edit) - it's whitespace, not newlines. I'm sure you've had files with spaces in the name.

in zsh you can use:

  P*~*wav*~*draft*

This looks a bit obscure due to lack of spaces, but it's simpler than it seems; the pattern is:

  [glob] ~ [exclude glob]

  P* ~ *wav* ~ *draft*

The ~ being the negate operator, which can be added more than once.

It's essentially the same as "grep -Ev" or "find -iregex".

It's a lot less typing than find, and also something you're likely to use interactively once you're used to it, so it feels very natural.

It's not necessary to bring Python into it, Bash can handle filenames with weird characters properly if you know how to use it.

E.g. instead of `ls | grep -Ev 'wav|draft'`, you'd have to do something like

    for filename in *; do
        if grep -E 'wav|draft' >/dev/null <<< "$filename"
        then : # ...
        fi
    done

Of course, it's more convoluted, but when you're writing scripts that might be used for a long time and by many people, it helps to know that it is possible to write robust things. Tools like shellcheck certainly help.

grep -q and you won’t need the redirect of stdout.

The above is perfectly fine for small directories, but in general the preferred way to loop over files is with find:

  find . ! -name . -prune                   \
           -exec grep -qE 'wav|draft' {} \; \
           -exec "${action}"             \; ;

Edit: I missed the herestring in the original code, so the above is wrong as mentioned in the comments; if your find has regex, you can use it to save one grep:

  find . ! -name . -prune              \
           -regex '.*wav.*\|.*draft.*' \
           -exec "${action}"        \; ;

Otherwise you can call sh to printf the filename into a grep.

However, the point of my post is that find can perform seek, filter and execute, and should be used for all three unless it is really impossible (which is unlikely).

Your example is grepping the file contents, where GP is using grep to select the filenames.

D'oh!

At that point I think you need to ask yourself why you're using Bash to begin with. If it's just meant to be a quick script that's run occasionally then this is good but probably overkill. If it's going into prod to be run regularly as part of business critical, then it should be in a language that has a less convoluted way to _ls a directory_. There's an inflection point somewhere in there, where it is depends on you.

Am I monitoring the execution?

Yes: bash is probably fine.

No: real programming language time.

Before you write anything, you need to think about the cost of it breaking and the chance of it breaking, and Bash scripts in VC tend to maximize both. I like that heuristic a lot.

Reply on news.ycombinator.com