test, [, and [[ (2020)
jmmv.dev
jmmv.dev
Here is something related from 2021 that also touches on bash's [[ operator and that I think you might enjoy in this context: https://jmmv.dev/2021/08/useless-use-of-gnu.html
In some non-bash shells, the `function` keyword is needed to declare certain types of function.
For make `$(shell)`, if you're building a lot of targets the performance difference can be measurable. Still, it loses in the nop case, so you should actually usually do `include` to trigger re-making.
GNU is completely right to ignore POSIX, since POSIX is not useful for solving most real problems.
In shells that aren't POSIX-compliant [0], maybe. In which case: yes, there are many wildly different scripting languages with REPLs.
[0] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
POSIX, being a standard that can be tested, is at least a specification or agreement that can be met by multiple products to enable them to interact
> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
:)
What it doesn't explain is why a Linux user should much care about portability. OpenBSD and FreeBSD are alive and well, but the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop? Do I also need to consider something obscure like vxWorks?
BusyBox (Alpine) is more interesting, but the changes there are so significant that a port will almost always be needed anyway.
Are there other compelling reasons to care about the non-GNU ecosystem?
----
What I'm trying to say is that open standards (and the portability that comes with them) is not something that just happens on its own. It takes active maintenance, and part of that maintenance is opting to adhere to the standard even when it would be more convenient to use extensions available in the most popular systems.
Will you personally suffer from liberally using Bashisms? Not in the first order. But if we encourage that sort of thinking as a rule, the standards become meaningless. I believe that would be a net negative change for the world, but there are many intelligent people who would disagree.
From my PoV it is generally portable anyway: almost everywhere I use a shell of that nature bash is present. The exceptions to this are things that need to run in small environments, like anything which may end up in initrd (where busybox us generally providing shell/script support).
Though I do wish people would make sure they specify bash in the #!/bin/bash in scripts unless trying to stick to the standard, as using #!/bin/sh causes problems when extensions are used and something lighter than bash is the default script runner (i.e. dash in Debian, and again busybox commonly in small environments). Only use #!/bin/sh if you lnow you are making an effort to be compliant.
This is the best argument I know in favour of not standardising, and the danger is not the direct consequences, but rather than our expertise in the standard atrophies when we do most of our work outside of it, which means our public work suffers too.
Also sometimes the private goes public and then who's gonna fund redesigning it in the standard? That rarely happens.
> From my PoV it is generally portable anyway: almost everywhere I use a shell of that nature bash is present.
Much like Internet Explorer 6, you mean.
A key difference is that bash is not claiming to be standard where it is not. It implements the full standard as well as its own extensions. It does not do standard things wrong¹ in order for its extensions to work².
I see posix-sh/bash more like javascript/typescript than good-browsers/internet-explorer. Which makes me wonder: is there a gnu/bash->posix/sh converter out there…
--
[1] well, there are a few oddities, like == behaviour in test which can cause portability issues because the standard doesn't specify that and people might accidentally use it when trying to be as portable as possible, but this also affects sh so is a general environment issue not specifically a bash related one. Also that doesn't break standard portable behaviour.
[2] or in the case of IE, doing standard things wrong, and its own extension wrong between versions too!
I actually feel triggered.
That was such a nightmare period in my web-dev career (pretty much the entire 2000's), it pretty much singlehandedly pushed me to find backend work as much as humanly possible
The dash shell is small and fast, but it does not allow any bash/korn language extensions beyond what was recorded in the POSIX.2 standard in the early 1990s.
Linux users should care because the Debian/Ubuntu family use dash as the system shell, so this problem is very real as many have learned.
> the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop?
This is the same argument that many companies and products have made over the years (and still do at times) for ignoring Linux. To have a Linux user use that same argument against OSes with even smaller userbases is kind of amusing to see.
For scripts though, sticking to POSIX sh often makes sense, yeah. You should at least be aware if you use of Bash-isms.
Tangential, but your blog software seems to have mangled the headings; it's serving eg:
make $(shell …) expansion
make $(shell ...) expansion
<h1 id=make-shell--expansion>make $(shell …) expansion</h1>
(Note: iOS probably displays … as U+2026 "…", which often looks identical to "...", but is actually a single non-ASCII character.)
I assume there's some semantic reason in which each is appropriate. I don't know what it is and am curious though!
if [ a = b ] || grep -q ^hello$ /usr/share/dict/words; then
echo "test failed and grep succeeded"
fi
Isn't that normal everyday shell use?
(And I might quote '^hello$' personally just on principle for having special chars, especially dollar, and to help syntax highlighters.)
if [ a = b ]; then
echo "Oops!"
else
echo "Expected; phew!"
fi
[ a = b ] && echo "Oops!" || echo "Expected; phew!"
[ "$debug" ] && echo "what's going on" >&2
----
And the fact that the if block tests a regular command means we can also do things like
if grep -q 'debug' /var/log/nginx/access.log; then
echo "Debug request found!"
fi
Something I have not yet bothered to figure out is whether I should write
[ $(expr 1 + 1) -eq 2 ] && [ $(expr 2 + 2) -eq 3 ]
[ $(expr 1 + 1) -eq 2 -a $(expr 2 + 2) -eq 4 ]
Not to be taken as a general rule though. I might be mistaken but I think that bash would parse the line as:
([ a = b ] && echo "Oops!") || echo "Expected; phew!"
illo@joe:~ $ [ "a" == "a" ] && >/dev/full echo "strings match" || echo "strings don't match"
-bash: echo: write error: No space left on device
strings don't match
illo@joe:~ $
illo@joe:~ $ if [ "a" == "a" ]; then >/dev/full echo "strings match"; else echo "strings don't match"; fi
-bash: echo: write error: No space left on device
illo@joe:~ $According to POSIX, the -a and -o binary primaries and the '(' and ')' operators have been marked obsolescent. See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t... under "Application Usage".
[ $((1+1)) -eq 2 ]There’s no need for test(1) / [(1) or conditional expressions ([[…]]) if you’re doing arithmetic:
if ((1+1 == 2)); then …; fi
if [ a = b ]; then
echo "Oops!"
fi
[ a = b ] && echo "Oops!"
$ bash -ec 'if [ 1 = 2 ]; then echo true; fi; echo $?'
0
$ bash -ec '[ 1 = 2 ] && echo true; echo $?'
1
Bash has "help test" for a quick cheatsheet.
The [ command is very old; it was already present in Version 7 Unix in 1979.
while [ 1 ]; do ...; done
while true; do ...; done
What's the reason for always using [ ?
(Making life easier for others and your self)
test -d /nix && echo "$_ exists" || echo "$_ doesn’t exist"
(Not POSIX-compliant. Documented in Bash and Zsh.)
Why not?
[ -d /nix ] && echo ... D=/nix test -d $D && echo $D exists … $ V=nada echo "x${V}x"
xx$_ is nto documented in the man page but is in the Info manual. It is not only set to the last argument of a command but also to the script name on script startup.
$ bash --version
GNU bash, version 4.4.20(1)-release (i686-pc-linux-gnu)
[ ... ]
$ cat underscore.sh
#!/bin/sh
# ^^ mistake here, should be /bin/bash
test -e foo && echo $_ exists
echo 'value of $_' = $_
$ touch foo
$ ./underscore.sh
./underscore.sh exists
value of $_ = ./underscore.sh
But at the interactive prompt:
$ test -e foo && echo $_ exists
foo exists
In the first place, I code in POSIX, except in the rare situation of making something that is strictly geared toward Bash.
Magic global variables subject to hidden side effects are garbage; I already feel dirty enough when I have to use $?.
This piece of crap also breaks under the DEBUG trap:
$ debug() {
> :
> }
$ trap debug DEBUG
$ test -e foo && echo $_ exists
debug exists
$ test -e bar && echo $_ exists
!1!
Sorry, I'm not going back to a 1978 way of writing shell tests, in order to use some broken magic variable.
while true; do ...; done
I might want to avoid an external process call just to create an unconditional loop.
$ echo $BASH_VERSION
4.4.20(1)-release
$ type true
true is a shell builtin
$ type test
test is a shell builtin
$ type [
[ is a shell builtinYes, `[` is a command, and has a man page and everything, but in a script it doesn't look like a command. It looks like something "special" is going on.
Whereas using `test` emphasises the point that you're just running another program. That all `if` does is check the output status of whatever program is being run, whether that's `test`/`[`, or `grep`, or anything else at all.
(Personally, I don't think that emphasis is necessary. But I've been using shell scripts long enough that I understand these nuances fairly well without having to think about them much any more. So I think that GP's point is a reasonable perspective to have, and worth considering.)
That said, [[ being guaranteed to be built-in certainly had its purpose at ages where shell script performance had any kind of relevance, and that was no so long ago.
ifif [[ x == 1 ]] thenthen
echo "x is one"
elseelse
echo "x is not one"
fifi
forfor x in 1 2 3 dodo
echo "x is $x"
donedone
whilewhile [[ x == 1 ]] dodo
echo "x is one"
x=$((x + 1))
donedone
untiluntil [[ x != 1 ]] dodo
echo "x is not one"
donedone
And if they really wanted [[ ]] to seem like syntax instead of a shell command, they could at least allow it to be used without spaces on each side of it like $(( )) or parens or brackets in any other language. And every other language lets you use as many redundant parens as you like for clarity, without running twice as many commands or producing a syntax error or weird unexpected behavior. But I don't think clarity was ever a design goal with Unix shell scripting languages.
Ahem:
if [ x = 1 ]
then echo "x is one"
else echo "x is not one"
echo "namely, it's $x"
fi
[ -n $FOO ]
[ -n ]
Remember to quote your variables!
Or use [ -n ${FOO-} ] which will replace the unset variable with an empty string.
You need double quotes.
$ printf "<%s>\n" alpha ${beta} omega
<alpha>
<omega>
$ printf "<%s>\n" alpha ${beta-} omega
<alpha>
<omega>
The general form is "${FOO:-default}" where default can itself be another variable or whatever string you want.
I usually prefer to set default values at the top of a script though using this idiom:
: "${FOO:=bar}"
local foo=""With colon, tests variable for unset or empty. Without the colon tests only for unset. It's POSIX too:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
If we have nothing in place of bar, they are effectively same.
I use this pattern all the time for variables that should be overridable by whoever is calling the script, ie. `FORCE="${FORCE-no}"`, overridable with `FORCE=yes foo.sh`. I'm not sure of any other way to do this.
(Yes, using getopt or other option parsing is better, but for my own scripts, env vars are just such a simpler way to pass options, and after 20 years of using bash/unix/linux I still can't write a getopt stanza from memory.)
: ${FORCE=no}
while getopts :hab:c o; do
case $o in
a) opt_a=x ;;
b) opt_b="$OPTARG" ;;
c) opt_c=x ;;
h) usage 0 ;;
?) usage 1 $o "$OPTARG" ;;
esac
done
shift $((OPTIND - 1))Heh, was that an attempt at sarcasm? That's super difficult to do from memory, at least for me. Maybe I'm not as good at memorizing things as you are.
There was definitely a wink component, but at the same time, I meant it.
Half the code is just case / esac syntax, which you can (i do) find plenty use for outside of getopts. getopts itself is… manageable. In the end, the code is such a strong pattern, it's basically a snippet. The lack of variation is what makes it kinda easy to remember. Or you could put it in your notes or a gist and copy/paste.
> That's super difficult to do from memory, at least for me. Maybe I'm not as good at memorizing things as you are.
Nah, my memory is shot. It's a matter of practice, and not being shy with man pages. Use it often enough, it's in memory; after a long hiatus, the details are a "/^\s*getopts \[" away (assuming your man pager is less).
I.e. user runs:
script --foo --bar=yes --no-xyzzy --uiuez"
opt_foo=y
opt_foo_given=y
opt_bar=y
opt_bar_given=y
opt_xyzzy=
opt_xyzzy_given=y
script: no such option: --uiuez
To add a new option, just define the variable. Done.
Check its value wherever needed, and possibly the _given, if the code needs to know whether it's working with the default value, or an explicitly given value. That's it.
Effectively strict mode for shell scripts.
Bash has some odd behaviors and footguns but it can also be surprisingly reliable and versatile. Ive caught some seriously messed up stuff that I couldn't figure out otherwise, using set -x. Also shellcheck will forcefully hammer good practice into your head and catch a ton of hangups that are hard to know before making the mistakes.
also setting backup variables is a good idea like: pictures="${pics_dir:-$HOME/Pictures}"
Or whatever the magic string is. Enable all the errors at the start of the script
[ "$FOO" ]
But at least it explains why you need spaces on both sides of the brackets.
It feels like some completely archaic concern to target the minimal common shell denominator.
Also, we'd have to build and maintain our own just for platforms missing it, which increases cost and complexity. It's easier, cheaper, and less error-prone just to use a minimum common denominator that runs on everything.
Additionally, when your customers have strict policies regarding how applications are approved for use and have to vet and test everything you give them, it's best practice to avoid installing anything that you can avoid installing. Doing that minimizes the effort and hassle for both the customer and the developer.
Build and maintain your own bash or install script? You don't have to maintain it, it's just building which is already provided by original source build scripts. You can install it as "mysolutionsh", nobody cares as long as you link it to PATH or reference it explicitly. Still easier than avoiding bash everywhere but in personal setups. Yours already familiar with it, that's a waste!
"Just install bash" is not always easy, and is not even necessary when posix shell can easily do what most people use bash-specific syntax for.
For me as a developer, it's not a big deal. You can do all the same things, just in a slightly different way, and you have a script that can run almost everywhere.
Add in the bootstrapping pain it imposes due to autoconf and other stuff, I think there are many valid reasons to avoid it and choose another shell that is more auditable yet still has just as many eyeballs on it (e.g., mksh - the default on Android; dash - default on Debian; BusyBox - every embedded system).
I write my shell scripts in sh, because it comes with every unix-like os by default.
I know that bash has more features, but I've never really missed them. I switch from sh to perl for more complicated tasks.
To each their own, right?
Shellscript has way too many idiosyncracies and weirdnesses that would have been beaten out of a proper programming language by now. (I know that talking about weirdnesses is amusing in relation to perl which also has a whole armload of them.)
PowerShell would like a word with you.
bash, for all its many many many flaws, is quite clearly a REPL first, PL second.
Also, the interwebs suggest Apple used to use tcsh as the default shell[2]; I don't know when they changed that, but it may have been after bash 4 released? (Thanks, 10.3 was in 2003, so several years before the license changed)
[1] https://www.theverge.com/2019/6/4/18651872/apple-macos-catal...
Apple just didn’t update. It took them years to finally switch to switch to zsh.
> The older bash is still available.
Aside from it being bash (a great reason not to use it as far as I’m concerned) it’s now a 17 years old version of bash.
I thought people liked macOs for its vintage feel? Remember a time when computers could only render a single menu bar in a fixed location, feel the experience of SYN floods, run a version of bash that is old enough to vote in the next presidential election.
Embedded applications come to mind as well.
need a script to run on many different systems and/or need to write a script to be managed automatically by a service account? probably you want a shell with syntax that is guaranteed to be the same on all your systems.
As many here have noted, bash isn't universally available, with another possible issue being OpenWRT devices. Stock/base images tend to use a Bourne-compatible shell, not full Bash. Though the latter's installable through opkg, for sufficiently small devices (typical of consumer kit), you simply won't have the space to install them.
There's also the slight PITA that Apple's OSX ships with a very old, pre-GPLv2 Bash, out of licensing concerns. (Apple is phenomenally averse to GPL-based code, much as some *BSDs are, such as OpenBSD.)
And if you're dealing with legacy systems (which tend to be extraordinarily and stubbornly persistently legacy), you'll often find that either bash isn't present or is quite dated.
I freely confess that I tend to write fairly recent-feature bash scripts myself by default, and appreciate many of the newer features. But if and when I am writing portable code, I'll go back to Bourne-compatibility.
But when writing system level code, an appreciation for standards and the very long tail of legacy standards and the limitations they impose is in fact a large component of professional maturity.
For the rest of us, we have learned the hard way, sometime repeatedly, portability and adherence to published standards matters.
And they all run bash. They all understand `[[`, they all understand `~=`
If any of them did not, then I'd ensure they do.
Yes, shells that do not support modern bash syntax ARE archaic, no matter who sits in front of the screen. And same as I consider it part of my job description to keep systems up to date with security patches, I consider it part of my job to ensure they use modern versions of contemporary and widely used scripting languages and shells. And that includes bash.
So no, relying on features that can be expected on a modern system is not a sign of inexperience. Same as a dev is within his rights to not write his python code under the assumption that it has to talk with a python 3.4 interpreter, he is within his rights to not expect a system that only knows sh.
https://zsh.sourceforge.io/Doc/Release/Conditional-Expressio...
Confirming my facts for this comment, #TIL that "dash" is the "Debian Alquist Shell", that is, Debian's "ash" shell:
<https://en.wikibooks.org/wiki/Guide_to_Unix/Explanations/Cho...>
zsh and ksh have it; in fact I'm pretty sure it originated with ksh in 1988 or earlier.
While zsh, bash, mksh, ksh93, probably others have it, sure. But many don't -- and not totally irrelevant ones either. Debian's default, dash, for example, does not support `[[`.
IMO, unless you're writing something like shell-specific dotfiles, avoid non-POSIX features.
It's usually pretty trivial to avoid them, especially if you're willing to call other mandated commands like awk, etc. But often, with a bit of creative thinking, most non-standard features can be replicated with some combination of `set`, separate functions and/or subshells.
Shell scripts, in general, have dozens of footguns, are pretty much impossible to statically analyze, difficult to make truly robust, and many of the shells themselves -- e.g., bash -- have huge, borderline inauditable codebases.
I can think of a dozen reasons not to write shell scripts. Yet still, there is incredible value in the fact that some form of POSIX-compliant/compliant-enough shell can usually be found on most systems.
All of that value goes out the window, though, the moment you start relying on non-standard features.
Sometimes all those costs are necessary for the task at hand. For example, the whole point of GNU Autoconf is that it runs on a wide variety of systems.
On the other hand, many programs are for in-house or personal use and will not run on obscure systems. The cost of writing for portability simply might not be worthwhile in these situations. And that’s ok, notwithstanding some conventional wisdom of “always try to be portable.”
If you're writing an installer script or whatever that's going to be run on $many computers that you don't control: sure, it's probably best to go to the extra effort to stick with POSIX sh.
But that's not most scripts. Most scripts are things that you run on computers you control. I just write things as zsh scripts because it's so much easier than bash (never mind POSIX sh). That's fine, because I can just install zsh. And actually, this often makes scripts more portable because you need to rely on external utilities a lot less.
I believe that [ should never be used, only "test" because [ gives the illusion that the mechanism is some part of the language syntax when it is just another "program" (I'm including built-ins and functions in "program").
(if / || / && look at exit status; a program can't see the exit status of other things other than looking at the magic variable $? which is just another string once expanded; case looks at strings but doesn't operate based on exit status and doesn't set an exit status as part of the case ... esac operation; "programs" set an exit status)
I also believe that [ / test should only ever be used for evaluating filesystem constructs -- test -f /dev/null and if you've got string evaluations use case.
Unsurprisingly, most scripts make me itchy, and scripts that I write people find weird.
[edited to add the explanation of "program" vs "syntax" opinion]
I prefer, when writing in shell, to do this:
if
program
then
something
else
otherthing
fi
if
ls /tmp/goober
test -d /tmp/goober
echo "I'll always execute the then clause because echo will always return a 0 return code"
then
echo "this always gets run"
else
echo "this never gets run"
fi
[edited to do code blocks]
if ls /tmp/goober
test -d /tmp/goober
echo "I'll always love you or whatever"
then echo "the grasshopper always jumps higher"
else "never gonna let you down"
fi
This works particularly well, visually, when nested ifs are involved.
I picked up the habit of preferring `test` when I was writing scripts that needed to run in both `sh` and `bash`, but I kept it because it makes more semantic sense to me than treating a character like `[` as a command. It's also weird that `]` is an argument to `[` rather than also being a binary. I mean, I understand the technical reasoning... but it feels like a hack.
yelp(){
es=$1
shift
echo "$@" >&2
exit $es
}
test -d /tmp/goober || yelp 33 "couldn't find goober!"
https://google.github.io/styleguide/shellguide.html#s6.3-tes...
In a scenario where you also need to support shells other than bash, Google’s shell style guide doesn’t say to use [ over test.
if [[ "${foo}" =~ ^bar$ ]]; then echo Yes; fi
85,000 lines of bash and counting... Not saying bash is great, but it's still meeting my needs for a lot of stuff. Shrug.
[ "$foo" = bar ] && echo Yes
[ "$bar" = extra* ] && echo '$bar began with extra'
1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...
My only guess as to why I am unfamiliar with the command is that perhaps younger me failed to figure out "expression" means regular expression. It only clicked this time because of your comment and the see also: re_format link at the bottom.
Personally, I prefer the "test" form to the "[" form. It's easier for me to read. Personal preference. To me, "[" looks uglier.
ls -li /bin/\[ /bin/test
26016 -r-xr-xr-x 2 root bin 133256 Mar 25 2023 /bin/[
26016 -r-xr-xr-x 2 root bin 133256 Mar 25 2023 /bin/test
$ docker run --rm -ti alpine
/ # ls -l /usr/bin/test /usr/bin/[
lrwxrwxrwx 1 root root 12 Sep 28 11:18 /usr/bin/[ -> /bin/busybox
lrwxrwxrwx 1 root root 12 Sep 28 11:18 /usr/bin/test -> /bin/busybox
/ # ls -li /usr/bin/test /usr/bin/[
495254 lrwxrwxrwx 1 root root 12 Sep 28 11:18 /usr/bin/[ -> /bin/busybox
495360 lrwxrwxrwx 1 root root 12 Sep 28 11:18 /usr/bin/test -> /bin/busybox
/ #In other distributions, test is a separate binary, and [ is a link.
It's possible your shell pre-empts this requirement and presents it as a proper syntax error, but it would have failed anyway.
# file /bin/test /bin/[
/bin/test: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6fe552c80ab0b3d3e60de2ab09167329e222eb67, for GNU/Linux 3.2.0, stripped
/bin/[: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=99cfd563b4850f124ca01f64a15ec24fd8277732, for GNU/Linux 3.2.0, stripped
#/bin/test --version
#/bin/[ --version
[ (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Kevin Braunsdorf and Matthew Bradburn.
https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;...
if some_command 1>/dev/null 2>/dev/null; then
: # A-OK
else
here_is_the_code_i_actually_needed
fi
(The command was “docker volume inspect VOLUME”, and in one place in my script I had to do things when the volume existed, and in other places I had to do things when the volume did not exist...)
if ! command …; then
do_needed_things
fiIt works for simulating very basic boolean expressions, but it gets ugly quickly if you need some more complex combos of NOT, OR and AND. I wish Bash had proper boolean expressions support.
What else do you need for "proper Boolean expressions support"?
Something like this won't work to become false:
foo=true
bar=false
baz=$foo && $bar
echo $baz
foo=! $foo
foo=true
bar=false
$($foo) && $($bar); baz=$?
case $baz in
1) echo false;;
0) echo true;;
esac
foo=0 # true
bar=1 # false
[ $foo = 0 ] && [ $bar = 0 ]
baz=$?
echo $baz
You could also use numerical expressions although I don't recommend it due to being even less readable (even more so if you have expression complex enough that you need to protect against overflow):
baz=$((foo + bar)) // foo && bar
baz=$((foo * bar)) // foo || bar foo=true
if $foo; then
echo yes
fi
You'd be amazed at how much of the world runs (just fine!) on bash conditionals.
Doesn't mean those are good or even okay or that people should choose to use them.
EDIT: this comment is a bog standard HN middlebrow-dismissal and the blog post doesn't deserve this for to be the top comment. It was morning and I was grumpy. I can't delete it anymore so would appreciate some downvotes.
If the answer is "no," that means JS has applications (e.g., servers and web clients) that bash can't be used for. And they're saying JavaScript is bad at them. Bash is arguably worse, but it's usually not an option in the first place so you don't get to complain.
Noo-ot really except in a Turing-tarpit sort of way. Like it or not, once learned (!) Bourne shell together with the traditional tools is a well-designed user interface and works even better for that than, say, Tcl[1]. Not an automation or scripting language, a user interface for all daily interaction. I would absolutely hate to manually sort through my files in JS, while in shell I can often do it faster than in a GUI file manager.
And, of course, it’s also a fairly strong contender as a programming language in the paradigm of many pipelined imperative processes—probably because that paradigm remains largely unexplored. I can only maybe name Icon as a viable competitor, and Icon’s also very nice. (Python, no matter how “inspired” it is by Icon, has traded its command of streams for more mainstream ease of use.) By comparison, the old complement of parallellized JS build tools (Gulp? Grunt? I forgot, it’s been a while) always surprised me with how awkwardly it accomplished shell-script-equivalent tasks.
To be clear, it’s not that Bourne shell is good and JS is bad. JS is a passable Fortran[2], while shell is at best a marginally functional one—there’s a reason Awk exists. But shell competes in categories that most other languages don’t even try to qualify for.
[Yes, I know about rc. I happen to think rc’s focus on one-level lists of strings (incidentally shared by Jam, an attempt at a better make) is a mistake, and more consistent arbitrarily-nested quoting, giving a “stringy Lisp” in the vein of Tcl, would be the way to go.]
[1] http://yosefk.com/blog/i-cant-believe-im-praising-tcl.html
[2] http://conal.net/blog/posts/can-functional-programming-be-li...
The paradigm is called point-free programming[1], right? I don't know how "imperative" is relevant here--you can rewrite most of the coreutils in Haskell, for example.
That means bash has applications (e.g., batch processing and file management) that JS can't be (easily) used for. And they never said bash is bad at them. That...proves my point?
The likes this gets is the design simplicity and uniformity (across processes), not how it looks and what it actually does.
It's not like the shell script designers sat down at a meeting and said:
"OK, it's very important that we don't want people using this language to implement all sorts of business rules and complex object interactions, because decades from now there will be invented an Ousterhoutian dichotomy and government regulations enforcing that developers should use other kinds of languages for that, because of the essential definition of what it means to be a shell scripting language, so we've got to come up with some way of punishing people who attempt to do that, and introduce obscure hard to spot bugs in their programs as a consequence if they have the audacity to do that, or even if their initially simple scripts later get more requirements and have to become more complex. Now let's brainstorm about how we can do that, and make sure the syntactic syrup of ipecac we come up to solve this problem fits in well with the rest of the language design by being totally off-the-wall and unlike every other piece of syntax in any other programming language including itself."
Then again, maybe you have a point, and they did do it on purpose, judging by how terrible the rest of the language is!
"Language Design Is Not Just Solving Puzzles" -Guido van Rossum
https://news.ycombinator.com/item?id=20672739
https://www.artima.com/weblogs/viewpost.jsp?thread=147358
>Summary: An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design.
>Some people seem to think that language design is just like solving a puzzle. Given a set of requirements they systematically search the solution space for a match, and when they find one, they claim to have the perfect language feature, as if they've solved a Sudoku puzzle. For example, today someone claimed to have solved the problem of the multi-statement lambda.
>But such solutions often lack "Pythonicity" -- that elusive trait of a good Python feature. It's impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn't translate into a simple test of Pythonicity. [...]
http://lambda-the-ultimate.org/node/1298
>Guido: Language Design Is Not Just Solving Puzzles
>And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable.
>The discussion is about multi-statement lambdas, but I don't want to discuss this specific issue. What's more interesting is the discussion of language as a user interface (an interface to what, you might ask), the underlying assumption that languages have character (e.g., Pythonicity), and the integrated view of semantics and syntax of language constructs when thinking about language usability. [...]
Ousterhout's Dichotomy is a contrived descriptive not prescriptive fiction, to rationalize the design of TCL after the fact. And despite its flaws and limitations, TCL is orders of magnitude better and more thoughtfully designed and purposefully thought out and internally consistent than any Unix shell scripting language.
https://news.ycombinator.com/item?id=9970505
https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy
>Ousterhout's dichotomy is computer scientist John Ousterhout's categorization[1] that high-level programming languages tend to fall into two groups, each with distinct properties and uses: system programming languages and scripting languages – compare programming in the large and programming in the small. This distinction underlies the design of his language Tcl. [...]
>Criticism: Critics believe that the dichotomy is highly arbitrary, and refer to it as Ousterhout's fallacy or Ousterhout's false dichotomy.[4] While static-versus-dynamic typing, data structure complexity, and dependent versus stand-alone might be said to be unrelated features, the usual critique of Ousterhout's dichotomy is of its distinction of compiling versus interpreting. Neither semantics nor syntax depend significantly on whether a language implementation compiles into machine language, interprets, tokenizes, or byte-compiles at the start of each run, or any mix of these. In addition, basically no languages in widespread use are purely interpreted without a compiler; this makes compiling versus interpreting a dubious parameter in a taxonomy of programming languages.
I was responding to the parent comment, saying "why do people like this". Its minimalism is ok-ish for small programs where you don't feel the pain so much and kind of beautiful in its solutions of packing everything into a separate executable.
And I absolutely stated that we have to implement the business rules and that I would be horrified to implement it in such a language. Therefore, yes, the language comes from a different age and shows it and, yes, would never consider it for any complex thing.
What annoys me is using JavaScript and shell scripts when there are clearly superior alternatives and no clear advantage for it besides the familiarity (which, admittedly, can be a strong argument).
Shell scripts being an arcane mess is no excuse for Javascript being as clunky as it is, and vice-versa.
I think a lot of this is a (li)nix culture thing.
Not liking javascript is "cool" in some sense. It's this new fandangled web language, not a "real" programming language in this culture. Shell scripting is not seen in the same light because its older and associated with unix I guess.
Not liking newer technologies in general is a thing that I have noticed, like with Rust or even C++ which isn't that new!
I do think shells have a better excuse for being bad programming languages than javascript though. Shells are primarily an interface to your computer and not a programming language. I use a shell in a terminal emulator to do most things on my system, like managing files and updating the system, you couldn't easily use javascript for this without writing a shell in javascript.
Shells are also much older than javascript, and "newer" shells like bash or zsh need to maintain some level of backwards compatibility.
Mostly shell scripts are used for very small tasks and not for writing large programs which is another factor.
You do bring up a very interesting point though. I think this idea applies to perl as well. It's interesting to see the difference in how these things are viewed.
The programs are fairly short, examples are common in the corpus, and the syntax and execution model are so inscrutable that only a machine can pretend to understand what’s going on.
Many people. It's not exactly rocket science to quote your arguments which already gets you most of the way there.
Hear, hear!
Honestly, I think writing a Bash script is just a terrible idea. Use a real language, and all of these nightmares go away.
For me, lately, that's been Zx (which uses Node), but there are other fine choices too.
I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html
The same can be said about QBasic as well, honestly. And here's the catch: people who are inclined to write anything in a principled manner are likely the ones who'd write things in a principled language instead of e.g. shell. Or to put it in another way, if someone chosen to write in shell (or failed to consider an alternative, which also happens quite often), they're quite likely exactly the person to not write anything in a principled manner.
Which brings me back to my point: yes, technically you can write decent code even in a sloppy language but that misses the bigger picture which is that most of the code written in a sloppy language will inevitably be sloppy because most of the people who write it won't care, due to the dynamics described.
Heck, the Ops department in my org was forced by the security guys to globally enable ShellCheck on commit for their internal repos, and those security guys still have to drop by about every 2-3 months to rip away their "# shellcheck disable" pragmas and force them to actually fix their broken code because those people in the ops actively don't care. The Ruby scripts they write end up slightly less broken because Ruby itself is a slightly more principled language, not because they suddenly care more when they write Ruby.
Not to mention myself: I've spent quite some time and energy on learning shell's semantics and tricks and quirks and DOs and DONTs but it's such an infinite, never-ending descent into abyss with rewards of dubious value that nowadays I just don't care. Whenever I have to write a one-off script, I give up even before I start and write it however sloppy, with minimal quoting, to save my time; and only when it breaks, or when I have to reuse it, or when I have to share it — which happens quite rarely — then I re-write it in a proper language. On the whole, that definitely saved me both my time and my sanity. As for the cases when I have to write properly behaving shell script, well, those are almost arise in the course of the tasks that can be delegated to our ops team and now that's their problem.
P.S. I found that re-writing naive but broken shell scripts in a proper language is quite easy: the intended semantics is generally obvious, it's just the shell that actually requires quirkier syntax to propely express it; re-writing the (mostly) non-broken shell scripts is much harder: you have to decipher the intended meaining from the quirky syntax while keeping in mind that the original author still could have gotten it wrong by not knowing about a particular quirk you're aware about (or vice versa).
Even carefully written bash code tends to be much less maintainable than code in better languages. It's just badly designed on multiple levels, like the famous post about PHP. Yes, if you're really careful you can write decent code in Malbolge - but why would you?
> I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html
The fact that you've written something that compiles to shell scripts rather than writing shell scripts rather undermines your claim that shell is a decent language. Integration tests of tools are important, but I'd still find e.g. TCL a much better way to write them.
So if you are extremely good at writing shell, and you write it extremely carefully, then you can achieve basically what you can do in other languages without those caveats?
It sounds like we both agree that shell is not a sensible choice for scripting, for most people, then.
It's a three (optional) steps thing really:
- I want something that can use on multiple systems right away (shell)
- Ok things getting a bit serious, I'll allow myself to add some pre-requirement tooling on the system (others script)
- pre-requirement sucks, I'll invest in native code tooling and produce binary (eg rust & go for the choices of the moment).