Common shell script mistakes (2008)
pixelbeat.org
pixelbeat.org
The best shell scripts are a list of imperative instructions to get something done. If an error occurs, bail as early as possible and with a useful message. Don’t try any harder than that.
For example:
if ! complex_function
then
handle_error
fi
You may have a clever way of solving this problem, but the cleverer your solution gets, the further you diverge from the core language of shell scripting. You will suffer the lisp/Ruby problem of every script being in its own unique language that’s based on but not identical to the language with which your colleagues are familiar.
One shouldn't dismiss shell scripts completely – the command line is the primary interface to Unix and with more and more idempotent commands showing up every year ("ip route replace") it only gets easier to write simple, imperative lists of commands.
I used to be an advocate for `set -e` etc but these days I've come to the conclusion that the POSIX standard just needs to be retired for simple shell scripts, even if it is just for personal use. And it doesn't have to be my shell language that replaces it, but it shouldn't be something aiming for Bash/sh compatibility. This is probably going to be an unpopular opinion though but I base it on years of Bash/sh use and then getting fed up with it enough to write me own shell + scripting language.
As for anything that needs to be used and maintained by a team, that's probably around the time you have to question if shell script is even the right solution and whether you need to break out into Python (though Python has it's own issues too so one has to make an informed decision based on your own business requirements).
In C we always write `if (func()) handle_error` for the standard 0=success convention, and reading shell IFs is mentally straining
My best shell advice is to put these commands at the beginning of every script:
set -e # stop scripts on errors
set -u # stop script if undefined var
set -o pipefail # stop script if there's a pipe failure
set -euo pipefail set -o errexit -o noclobber -o nounset -o pipefail
shopt -s failglob inherit_errexit
Example for set -e: https://mywiki.wooledge.org/BashFAQ/105
It made sense: everyone could read R, not many could read bash/shell as most had learned R on a windows PC in grad school of statistics.
So while sometimes a bit clumsy and not very portable, these scripts could be read by anyone.
Most linux distributions come with python installed. Anything more than invoking a binary and redirecting the output? Just write it in (pure) python I say!
Edit: by pure Python, I mean don’t require any `pip install`s
I agree that 50 lines of bash is generally more maintainable than 200 lines of Python.
I've never had subprocess deadlock — was this on Windows?
I've seen this happen seemingly randomly in linux trying to pipe the stdout of one subprocess into the stdin of another.
Compared to general purpose programming languages, it was built for "DevOps"y use cases in mind. See https://github.com/ngs-lang/ngs/wiki/Use-Cases
Ability to run external programs is high on the priorities list. In NGS it means having its own short syntax and handling exit codes among other things. NGS throws exceptions on "bad" exit codes. Hint: not for all tools non zero exit code is "bad". Did not see equivalent exit code handling anywhere else.
Compared to bash... It's not fair comparison even. Another era, another reality, other expectations. Small example: When APIs return structured data, well... you better be able to handle it. See https://ilya-sher.org/2018/09/10/jq-is-a-symptom/
Compared to other modern shells, I would say, NGS is programming language first as opposed to shell first.
Too many times I didn’t move from bash until too late.
I still write bash, I’ve got one which runs up a 20 line ffmpeg command with a couple of variables, that’s not a problem. On the other hand I changed the complexity of a file analysis tool from grep/sort/uniq/sed to Perl a couple of weeks ago before it became too large.
Couple decades ago even wrote some init scripts in Perl because they needed to be complicated.
Seeing that Ansible is basically a set of Python libraries that enable users to write declarative (and hopefully idempotent) scripts, it isn't a stupid question.
As a side note, Bash + Ansible tends not to be an amazing combination, unless you can write idempotent Bash scripts, IMHO. That sounds like a recipe for false safety, where you think you wrote strong automation, and someone that is used to pure Ansible will quickly realize that it does not work in specific conditions, e.g. after second time or after specific conditions are run multiple times in a row.
Shell script have their deficiencies but it's the best for their intended environment. Alternatively now there's Oil shell that looks promising for modern approach to shell programming [1]. Expect to see exponential usages of shell programming now that we have new popular OS related technology including containers, isolates, and functional package management like Nix and Guix.
If you don't know what you are doing, and shellcheck knows more, it might be useful. But if you know what you are doing, shellcheck becomes annoying quickly.
Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions.
I've begun piping such untrustworthy commands to xargs -n1 to guarantee good behavior.
It should still be `printf '%s\n' "$(command)"` for portability unless the intention is for the output of your command to possible be interpreted as a flag for echo.
Damn that’s ugly though. Not much better than Perl in terms of characters soup. And Perl has a massive advantage with its deep integration of regexes.
I can forgive a lot of design flaws because the tech was new and people didn't know any better but I always struggle to apply that same reasoning with the design of having commands that support both free text and flags that change that command's operation. I'm surprised nobody at the time paused and said "this might be a problem" when they were writing their tools and only using the hyphen to distinguish between a flag or data. Maybe I'm being a little unfair though -- it is hard to put yourself in their shoes and unlearn 50 years of tech.
My biggest peeve about it though is that I can write a shell that doesn't apply to POSIX and which fixes a lot of shortcomings of shell scripting. But I cannot fix this specific class of bugs without rewriting the entirety of coreutils.
- it is a GNUism so not available on all UNIX-like systems
- it is something you pro-actively have to remember to put in
- you cannot simply add `--` to every command because those that do not support it would fail (or worse, not fail in unexpected ways)
In an ideal world operators should be out of band from data (murex does this with `config` https://murex.rocks/docs/commands/config.html whereas other commands might solve this with environmental variables. Neither are as convenient as flags though).
In a less ideal world data should be encapsulated inside operators (see below), but this breaks globbing.
command --flag1 --flag2 --data=file1 --data=file2 --data=file3
Maybe if `--` was a standard from the beginning to act as a separator between operators and data (and enforced too, ie commands would fail if `--` wasn't present) then I'd be more forgiving for `--` as a solution.
It's a general lesson for linters: if it points out code where the results may be surprising, then be aware that the "surprising" result may be necessary for the code's function.
In Deployment from Scratch I teach minimum amount of Bash to get servers up and running. I avoid teaching anything I don't have to. For example, most people are fine with just "set -euo pipefail" and understanding simple functions and pipes.
If your script is small enough, you will be fine. If your script is getting big, it might be a time to switch to smth else.
cleaning up temp files
Use TmpFile. It is automatically removed when the script exits.
stopping automatically on error
As in many other modern languages, NGS has exceptions and prints stack trace when they are not handled. Nothing special is required from the programmer.
echoing errors
Use built in error("my message") function. Alternatively, use exit("my message") to print the error and exit, the exit code defaults to 1 but can provided. Tired of seeing bash scripts starting with definition of warn(), debug(), error(), etc functions. These are part of standard library in NGS.
You are welcome to check out the Next Generation Shell project.
I always go with #!/bin/bash as I do often use bashisms, and only #!/bin/sh when I know I've been careful (because at the start I know I'm writing something intended to be as portable as practical, or I've gone through with a fine-tooth comb to test compatibility at a later time).
I believe there are some esoteric systems that relocate "env" as "/bin/env" or such, but the good ole "90/10" rule applies here
Because the backwards compatibility tends to be good, if I'm writing a script that will be run non-interactively, I will usually write a fixed `#!/bin/bash` shebang just so I can be sure that it run with the expected bash version on Linux or Mac.