How to write idempotent Bash scripts (2019)
arslan.io
arslan.io
Never use potentially dangerous commands (rm -f) to avoid the prospect of an error!
Instead, one should practice good error handling. For some scripts and depending on the audience "set -e" may be sufficient. But usually always better:
rm file.txt || echo "Warning: Deleting file.txt failed, continuing anyway."
Or, for longer shell scripts, I always include a little function that deals with error handling. You can call it in the same way:
rm file.txt || errorHandler("ignore")
As shown above, the function can be built to take various parameters, and can then for instance abort or ignore the error. It can also take care of updating a log file.
Somewhat related, I do logging in shell scripts with a function, too:
rm file.txt && log("info", "file successfully deleted")
That way, I only need to write the code that writes log lines with nice formating and timestamps and such once.
Somewhat related, one can trap various signals in a shell script, and call a bash function when they happen. Thus, one can trap SIGEXIT and call a cleanup function that triggers on Ctrl-C, and I'm almost certain (sorry, can't test right now) one can also trap SIGERR and catch errors in a shell script nicely (though I'm not sure if that also goes for external commands inside said shell script).
errorHandler("ignore")
errorHandler "ignore"Between
[ -f file.txt ] && rm file.txt
rm -f file.txt
Although it should be said that concurrency and idempotency is seldom acquired just by using force-flags in isolated spots, the whole operation needs to take it into account. There are probably more steps involved than just removing one file. The linked article leaves out the bigger picture so it's probably not very helpful. Often it is sufficient to construct the new state under a temporary name and then move it in place in one operation.
unlink ('file.txt') or die "Can't remove file.txt: $!";Interestingly though starting in Bash was a good MVP approach. Fast to get started. But as things started to slow down the project switched to a better tool. I expect for many projects the number of bash scripts may never grow large enough to merit a switch. So, sure write scripts that are safe to run multiple times, but if that effort grows know that there are better tools.
I could be wrong though, the only reason this stuff sticks in my mind is because i was working on a CFEngine deployment (scarred for life) at the time Ansible popped up.
commit f31421576b00f0b167cdbe61217c31c21a41ac02
Author: Michael DeHaan <michael.dehaan@gmail.com>
Date: Thu Feb 23 14:17:24 2012 -0500
Genesis.
diff --git a/README.md b/README.md
new file mode 100644
index 00000000000..60bbc9f8137
--- /dev/null
+++ b/README.md
@@ -0,0 +1,88 @@
+Ansible
+=======
+
+Ansible is a extra-simple Python API for doing 'remote things' over SSH.
+Moving toward "declarative state with enforcing/monitoring (idempontent!?) control loops" is much, much better. (What k8s does well. Now all we need is one master bash script to setup k8s :D )
https://docs.chef.io/chef_solo/
While its a matter of opinion and will depend on problem you are trying to solve, I've found teams get productive on Ansible a lot faster and Chef Cookbooks often end up requiring significant maintenance. I have similar complaints with Puppet too.
My biggest complaint with Chef is that its DSL so closely resembles Ruby that developers often assume the code in the cookbook is executed as pure Ruby rather than a DSL conversion; the transcoding phase in Chef is pretty complex and can lead to lots of debug head scratching.
And it's malleable like clay, you can turn SaltStack into a perfect fit for your case
> rm -f example.txt
Instead of using --force to clobber all sorts of permission scenarios, wouldn't you want to specifically avoid non-existing files and handle that? I'm thinking something like....
stat -q example.txt && rm example.txt
Resolving the race robustly is actually kind of hard, in Bash... You can check whether `rm` returns a nonzero exit status, but that might fail for all sorts of other reasons (e.g., bad permissions). I guess you could case/select the exit status numerically, but that could turn into a real case of bedbugs, really quick.
I guess you could repeat the `stat` check afterwards, if the ~`rm` fails? In that case, if the path doesn't exist afterwards, you can just swallow the `rm` error and let it roll.
Practically speaking, I try to avoid using Bash for anything where automatic, reliable error handling is that important... I love cool Bash tricks, but it's just not designed for sophisticated control flow.
`rm -f` handles some errors in a different way than the stat/rm approach. One can fail where the other would not.
Any given unhandled error may or may not be a problem, depending on the nature of the error, how the rest of the script is structured, and what your requirements are. There's nothing wrong with the stat/rm approach--it may be the better way to go.
That just means you have to do a little more work.
1. Become root.
2. Fork until the process table is full.
3. Kill all other processes. After each kill, fork until the process table is again full.
4. When all processes other than the init process are yours, you can do the stat and unlink without worrying about race conditions since there is no one else to have a race with. (Assuming the file you want to stat and unlink isn't some file that your init process is interested in, and assuming it isn't on a network drive).
The side effects of this are annoying to deal with though and are probably worse than whatever problems an unhandled race condition would cause.
Another approach would be to create a new user, chown the directory containing the file to that user, become that user, chmod the directory to 0300, kill any process that has that file open, do your stat and unlink, chown and chmod the directory back to what they were, and delete the earlier created user.
I just reboot in single user mode.
(Or, if anyone wants to get pedantic and yes, this is ruining the punchline, but alas: shutdown -h now ; then reboot from the USB stick and mount the partition ; rm file.txt).
If your script has to be idempotent in the face of concurrent executions of itself, then that's an extra requirement which you have to handle with locking or whatever.
It is not implied by idempotency; there is meaningful idempotency which excludes the concurrency requirement.
If something can rename example.txt in parallel with your script, at any time during its execution, there is nothing you can do to ensure that it's gone.
Whatever step you take to ensure that its is gone can be preceded by the parallel rename.
I believe your point about idempotency is correct, though.
https://www.gnu.org/software/coreutils/manual/html_node/stat...
(At time of writing, GNU Coreutils 9.0)
Though you're unlikely to be writing multi-threaded bash, but entirely possible for another process to delete it too.
For shell scripts, a reasonable approach is to use a (correctly implemented) lock file at the start, then a bunch of `test` (aka `[`) blocks to check things. That does a decent job of documenting, too. Vs. everyone who looks at the script needing to know the less-common command line options.
Then, in the bigger picture, make sure that nobody else's script - maybe with its own lock file - starts overlapping with your script.
maybe, depending on your definition of idempotent. It does update the modification time of the file, which does result in a different state if it is run multiple times. Although in practice, that probably doesn't matter most of the time.
Also, if you have a complex firewall then putting your rules in your own chain and inserting that chain into a top level chain is the idempotent way:
iptables -D FORWARD -j LOL || true
iptables -F LOL || true
iptables -X LOL || true
iptables -N LOL
iptables -I FORWARD -j LOL
Much nicer than polluting the top level chains with the iptables equivalent of global variables. You can work on your own but if the firewall without clobbering all the other stuff.
e.g. Approx zero percent of the stackoverflow codesnippets you google are idempotent.
So as a new learner that's a show stopper right there. Barely hanging on to the stuff you're being taught let alone being able to see a big picture concept that is 20x miles above your current level.
Its just not happening
I thought the big task would be introducing source control and CI/CD, but I quickly realized idempotentcy is actually the more fundamental and key concept that I need them to embrace.
The example checks for /mnt/dev in a file but it doesn't check whether the string is in a comment or if it's part of /mnt/develop, and pretending you've achieved idempotency is dangerous unless you've gone through the effort of checking every corner case.
It's easier and safer to pretend it's not idempotent in the first place.
Very interesting!
(Of course a few years ago immutable infrastructure was the rage, because it means if the script runs once, pack it up as a VM and you're done. And that's how the docker sausage images are made.)
Been looking for one a while a go and it feels like it’s something sound be built in.
It's basically meant to look up spellings in /usr/share/dict/words, but it can work on any file. It will match any line that your pattern is a prefix of, so you'd have to add logic to eliminate longer matches.
But if you had some huge file to search and you wanted to do it from a shell script, that would be one way. Caveat: although it's fairly standard, 'look' might not be installed on every system.
Also, you have to be sure to maintain your file in sorted order. So no adding things by appending to the end, and checking if something is in the set is much quicker at the expense of adding things being much slower.
> Return the unique lines in the file words.txt that don't exist in countries.txt
comm -23 <(sort words.txt | uniq) <(sort countries.txt | uniq)
comm -12 <(sort words.txt | uniq) <(sort countries.txt | uniq) ~/bin/union
===========
#! /usr/bin/awk -f
!acc[$0]++
~/bin/intersection
==================
#! /usr/bin/awk -f
!buf[$0]++ {acc[$0] += 1}
ENDFILE {
delete buf;
files++
}
END {
for (k in acc) if (acc[k] == files) print k
}
~/bin/set-diff
==============
#! /usr/bin/awk -f
! filenum { acc[$0] = 1 }
filenum { delete acc[$0] }
ENDFILE { filenum++ }
END {
for (k in acc) print k
}Just because you can't reach perfection doesn't mean it's useless to strive towards it.
Idempotency is a very useful property that more shell script writers should be made aware of, even if they only learn basic things like "mkdir -p" or "rf -f SOMEFILE"
Obviously idempotency is a powerful tool, but the confidence with which tfa demands it of all software was silly.
mkdir dir
chmod a-x dir
mkdir -p dir
I've come to the idea that the best thing to guarantee what you want is
mkdir the-new-dir.temp-jnsjw23
<do things to the-new-dir.temp-jnsjw23>
rsync -a --delete the-new-dir.temp-jnsjw23 the-new-dirThe effort to get there also shows you how elusive idempotency can be, especially if strictly defined.
Another thing I would recommend is usage of temp directories: you can assume they are empty (easier to reason about). Then do all the work there, an when all operations were successful, you "commit" modified files with "mv/cp -r" and overwrite the real files.
I would call the use of temporary files/folders "transactionality" rather than "idempotency" though.
I was confusing the order of LN arguments for years until I read a pro tip that they are the same order as CP arguments.
cp file1 file2
And now
ln -s file1 file2
No more confusion.
cp --symbolic-link file1 file2
cp -s file1 file2Apart from doing what the article suggests, I make sure that every step of a process is rerunnable by cleaning the state beforehand.
E.g.
rm -rf /path/to/dir mkdir -p /path/to/dir # use the dir now
If a command doesn't have the right switch to handle both cases (like using semanage), I use || to fallback to another command.
It feels pretty manual compared to some declarative approach, but it works nicely in small Bash scripts.
For lots of people, bash is the hammer, everything's a nail. Can be very frustrating to inherit such environments.
Maybe you don't need portability in your code, but maybe you could use more portability of your skills.
Maybe you can assume that ansible will be on every system you admin, but maybe that will definitely reduce your options and it's possible you don't even know what benefits you are trading by not having the depth.
It's all a big maybe, but maybe some of those people made good trades for the problems they encountered.
> If this is run again, you’ll end up having duplicate entries in your /etc/fstab
You could also just add a space in front of a command you don't to accidentally run later. This will not store it in .history
totally foolproof except for the race condition between when your script crashes and you hit up-arrow then enter, someone else with your user account might change it maliciously.
Better to use Python, or Ruby, or a Rust or Go executable.
It just doesn't.
Edit: oh and you can write non-idempotent ansible playbooks too... it's not about the tool, it's about the craft, and experience.
And passing force flags can lead to subtle issues as -f and friends may have wrong behavior in corner cases. So for this reason I do not use it in shell scripts and rather do explicit tests before the command, like test -f file && rm file
Also for systems stuff you’re not gaining much lot when your interface is calling binaries. A script of subprocess.run calls is way more cumbersome and the moment you want to pip install something it’s no longer portable and a huge PITA to distribute.
Bash has lots of footguns because it’s been around a while; run your code through shellcheck if you’re not confident. It’s the lingua franca of Linux userspace.
There will be religious wars until the end of time about Python vs Go vs Ruby vs Rust vs "The New Hotness". Popular languages change; bash, as a shell language, outlives them.
In 1995 Perl was the hot language. Bash was there too. Now people can't maintain Perl code, but Bash still runs. Bash doesn't require add-on libraries or modules, it uses system built-ins. It's the lowest common denominator for system scripting.
Also, while Mac tries to force people to switch to zsh, and most lemmings just obey, you can still make Macs obey you, and use bash as your login shell.
... which themselves differ by system.
You have a flawed understanding of Mac users, and of lemmings, and of zsh.
Given the prevalence of git in the software development world, this has meant in my experience bash has very often been the most effective "zero dependency" cross-platform scripting language around. Python/JS/<other runtimes> will rarely work out of the box 100% of the time especially for Windows developers. This makes bash pretty valuable for things like build scripts if you have developers using different OSes.
If your source lives in git, you know there is a good chance user must have gitbash.exe too if they managed to clone a git repo on Windows, thus granting as near as one gets to "zero-dependency" cross-platform scripting in my experience.
So my rule of thumb if one cannot use plain /bin/sh and must depend on bash-specific things, one better use a proper scripting language.
But generally not with stuff that one typically uses for scripting. In most ways, it's a superset of bash.
Besides word-splitting on unquoted variables being turned off with default settings (which is turned on when zsh is called as bash/sh with a symlink or similar), what other differences do you think would be common to find when running random bash scripts with zsh?
The subject is bash scripts, so it’s safe to assume anyone reading the article has decided they have a good reason for implementing the work in bash. As someone who is trying the level up their bash skills, I found the information in this article very useful.
If you think Python is the answer then you have to deal with the whole nonsense around pipenv/poetry/pyenv/asdf/virtualenv/setup.py/requirements.txt/pyproject.toml/wheels/... 5 hours later .../system packages/anaconda.