A common mistake involving wildcards and the find command (2020)
blog.robertelder.org
blog.robertelder.org
Thankfully most languages didn't make this mistake but some did: CMake, YAML, Bash, TCL. All notoriously awful.
As well as Perl and PHP, though both have since recanted (Perl started 20 years ago with strict mode, which IIRC forbids barewords in most contexts; and PHP deprecated barewords in 7.2 and removed them in 8).
Could you elucidate?
The biggest problem I see with people dealing with Tcl is that they carry their understanding that {} delineates scope (or something else lexically). Of course, this is wildly untrue in Tcl as {} is simply a "quote" operator.
This stuffs people up horribly until they unlearn the idea that {} has semantic meaning.
puts Hello!
puts "Hello!"
puts {Hello!}
# or even
{puts} {Hello!}
# x is not actually a list, but may appear that way if a and b are nice
set x "$a $b"
# Proper way:
set x [list $a $b]
"Sincerely," "ls" "-lah" "foo" "bar."
Alternatively it could let you type without quotes and then automatically quote it for you (e.g. you press shift-enter and it auto-quotes it).
It's fine for interactive use because you can debug errors when they happen. The issue is with scripts.
The problem with “works most of the time” is that it stops working when you’re not around. So it might be fine for a one-off shell command, but you never know what the next guy will pass into it.
Generally a good idea if you care about bugs at all.
Shell is a REPL that has some special filesystem-related helpers. Looking through $PATH for any +x file that matches the name of the first word entered. Performing glob expansion (*, ?, {foo,bar}...) on all but the first word. Backticks. Environment variable scoping (difference between `export FOO=bar` and `FOO=bar ./my_program`). Presence of "." and ".." in the FS. And so forth.
Once someone wraps their head around the execution model, they should be golden.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
On systems that can support it, there is an additional expansion available: process substitution.
Furthermore, expansion absolutely does work on the first word and it's not obvious to me why someone might think it doesn't. The simplest example is probably tilde expansion, i.e. the ability execute ~/bin/foo or ~someone/bin/bar. But the other types work too, e.g. brace expansion:
$ ls l1
ls: l1: No such file or directory
$ l{s,1}
ls: l1: No such file or directory
$ touch l1
$ l{s,1}
l1
$ $(echo l)s -l l1
-rw-r--r-- 1 user wheel 0 10 Dec 14:12 l1
$ perl$((4+1)).$((36/2)) -E 'say "foo"'
foo
$ ls -l
$ touch find foo bar quux
$ fi* . -name f\*
./foo
./find
As was already pointed out, you are right. It does not work to match executables in the $PATH, which is what I was, for some reason, trying to communicate badly.
Since we are being pedantic regarding the terms, while I can't really find "glob expansion" in the Bash manual either, it documents a bunch of "glob" settings: "noglob", "GLOBIGNORE", "dotglob", "extglob"...[1]
FWIW, I expected to find these in the Bash info page on Ubuntu, but curiously, "info bash" does not show up the info page (just the fallback manpage) — "info find" does work. Whatever happened to (tex)info pages?
[1]https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash....
From the bash manpage (version 5.0):
The special pattern characters have the following meanings:
* Matches any string, including the null string. When the globstar shell option is enabled, and * is used in a pathname expansion context, two adjacent *s used as a single pattern will match all files and zero or more directories and subdirectories. If followed by a /, two adjacent *s will match only directories and subdirectories.
The GNU bash manual describes the '-f' option as "Disable filename expansion (globbing)."
"Globbing" is how * expansion is generally referenced in other documents, if not often within the bash manpage or info pages themselves.
It is because there are so many layers. The way commands are parsed is already not that easy, but then the command itself has its own rules, and it is not uncommon to pass a script as an argument.
for example if you type "sh -c ls", first it is parsed into exec("sh", "-c", "ls"), then "sh" parses its arguments "-c" and "ls", then a sub-shell is started that parses the command "ls", then the actual "ls" program is launched.
This is simple but at every step, there are some quirks, first is all the shell parsing issues, with globs, interpolation and all that. Then you have the issues of how the command parses its arguments (be careful about "-"). Then back to the shell parser with the sub-command, then "ls" itself has its quirks, like locales for instance.
I have done Perl (line noise), C++ (memory unsafe with too many features), PHP (injections everywhere), JS (with IE6 support) and none of these languages make me more uncomfortable than UNIX shells.
Overall Windows command line is and always was broken and unsound due to being based on the flawed DOS batch model.
Informative writing gets to the point instead of playing gotcha games.
find . -name %.jpg
A more intuitive solution would be to resolve globs to nothing (the empty string) if they don't match. In most users' mental models "*.jpg" means "any file ending with .jpg" when they pass it to a command like "rm" or "mv", but in reality it means "any file ending with .jpg in the current directory, or the literal string '*.jpg' if there are none" which is nonsensical. If it expanded to nothing, it would be obvious that it needs escaping when passed to commands like find because find would complain about not being passed a file name. The failure case is outright user-hostile and error-prone.
Also if every tool used its own metacharacters, users would have to re-learn them for every tool. It's common for config files to also use globs, for example. Even if we just had one set of characters for shells and one set for other tools, it's often difficult to draw the line, especially if tools use shells and thus would need to translate from one set to another. If this is a design error in find, it's a design error in all tooling that isn't a shell and we need to throw it all out and start from scratch.
~ echo f43f3f*gsdgds
zsh: no matches found: f43f3f*gsdgds
~ bash
user@debian:~$ echo f43f3f*gsdgds
f43f3f*gsdgdsI agree, so I use shopt -s nullglob in both bash scripts and interactive shells. It retrained my interactive habits with find very quickly because unquoted wildcards no longer work in the common case where there's no match in the current directory.
(I like dotglob, extglob, globstar and pipefail everywhere too: saner defaults than emulating traditional shell behaviours.)
> failglob
> If set, patterns which fail to match filenames during filename expansion result in an expansion error.
> nullglob
> If set, Bash allows filename patterns which match no files to expand to a null string, rather than themselves.
[1] https://www.gnu.org/software/bash/manual/bash.html#The-Shopt...
I don't disagree with your central point that this problem can be fixed with better design. But the ergonomics are a hard problem to solve.
The issue with find here is that globs act consistently, but the returned behaviour is dependent on when the evaluation is performed.
find(1) treats * just as the shell does, but only if the shell hasn't got to the * first. So to prevent the shell from pattern-matching any * characters, you've got to quote or escape them using single quotes (') or backstrokes (\).
Quoting to prevent interpretation of metacharacters is actually commonly encountered in computer contexts. I'm having to quote instances of * but not \ for example in this comment to HN as both have semantic meaning to the content parser. Note that the escape character '\' is only significant when used immediately preceding a *.
IOW, I hope all your find deletes are preceded with a run without the -delete flag. If you get into the habit of doing that and then doing any destructive action using a separate process (piping into xargs), you'll be spared from any hair pulling.
My preference is for any complicated shell magic to first do a run which performs `echo` of whatever operation I was doing on a file (eg. `find | xargs echo sed -i 's/foo/bar/' $file`), but there are other ways to achieve the same thing.
Another tip for anyone who runs history commands like `!!` or `rm !$` is the :p operator, which just prints what the shell would have executed and adds the full command into your history so you can hit ^ enter.
The echo trick is mostly a suggestion of a very quick trick, but it fails in a large number of cases (even something as simple as piping that command through another).
Thanks for the pointer to :p too!
Don't surf from the bed, get up, do something, and get back to the bed when you are ready to sleep. (nope, I am not your wife even if I sound like her :)
Now that I've fixed your problems, how do I get back to sleep after I wake up or get woken up at 3-4am? :)
Medito (https://anagora.org/go/medito) is a good free and open source app for mindfulness meditation, produced by a nonprofit.
(I hope I can do away with the phone for it soon enough, or it may turn out to make things worse :))
Honestly, I just run 'find . | grep' to achieve what I want these days.
fs() { find -iname '*'"$1"'*' ; }However even without newline it will be fun to process all the special symbols in file names if you decide to handle somehow the matched files.
PS Unless I missed /s in your message, which I'm not always good at :)
A common mistake involving wildcards and the find command - https://news.ycombinator.com/item?id=22278602 - Feb 2020 (143 comments)
FWIW, just seeing the title, this is what I expected to read.
And even allowed things like 'RENAME *.txt *.bak'.. one of the most surprising things when starting to use Linux was that there was no simple way using the default utilities to accomplish the same thing...
Unix/Linux takes the view that the shell is responsible for interpreting parameter expansion, and that once expanded, those parameters are then handed off to the invoked command. Yes, there are edge-cases, but they are the same edge cases for each command. There's only one set of knowledge to internalise.
(I began that internalisation some 35 years ago and am tremendously grateful that much of the knowledge has been cumulative rather than substituting for earlier methods. The areas I consistently have the greatest challenges are those in which behaviours or methods have changed (in cases multiple times) over the years and/or decades. Especially for tools which are seldom used.
The bash shell isn't fully consistent, I'll grant. But it's also what I'm using much of the day, every day for interactions, and even its inconsistencies become hardwired over time.
% echo no*such*file
echo: No match.
Whereas in bash you can get used to this behaviour until it bites you. Luckily the tcsh usage has already given me the "finger memory" to use the quotes.
$ echo no*such*file
no*such*file