Writing Safe Shell Scripts (2019)

Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed).

1: https://www.oilshell.org/

https://github.com/mvdan/sh

I'll probably get flamed for this, but Python is an absolute fucking trainwreck of a platform. Even at its core the 2.x vs 3.x is going to break all kinds of stuff in Jan 1, 2020. Just look at the package system: cross-system portability is a mess (don't believe me? run on Arm for a week), and unfortunately because there IS a package system people think they HAVE to use it, which means: BLOAT BLOAT BLOAT your BOAT, gently down the stream!

It's kinda like what Dewalt did to miter saws. We've gone from something that could be kept perfectly square and did a great job and took up a little bit of space, to 30+kg behemoths with sliding rails and double bevels, tons of blade travel and deflection, and are a challenge to keep true and take up 3x the space and weigh a ton.

Keep your mission-critical tools simple. Period.

PR's welcome. ;)

Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash.

Here's an example. I use `llbranch` all the time: https://github.com/shawwn/scrap/blob/master/llbranch

It lists all branches in a project. It's nicely colorized. What would the equivalent Python be? Who knows, but it'd be much longer.

https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scrip...

There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing shell scripts and there's never been a Python ad claiming that just anyone could write it.

Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV.

Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The two Pythons are basically incompatible unless you're willing to put in an effort. You have to work to make Python scripts compatible with both versions, especially if you can only use built-ins, and the results are way clunkier than that 2-line shell script you're trying to replace.

Python is a terrible choice if your goal is a relatively short script that calls other programs. You can call out to other programs in Python, but it's much clunkier and more painful to maintain. And while it often doesn't matter (because neither are speedy), CPython 3 takes more than 50x more time to start than dash and 27x time to start than bash (see: https://github.com/bdrung/startup-time , which shows Python3 3.6.4 at 197.79 ms, Bash 4.4.12 at 7.31 ms, and dash 0.5.8 at 2.81 ms). That speed is helpful when you want to have a "simple script in a loop".

Python is much better than shell when you start needing more sophisticated data types, libraries, modularity, etc. But the typical use for a shell script doesn't need any of that. If you need that, shell is a terrible tool, because it's the wrong tool for the job, not because it's never useful.

If you're writing shell, use shellcheck. Once you do that, your error likelihood goes way down. Many of the reports of problems writing shell come from a time when shellcheck didn't exist.

The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time.

A recent example was I needed to tar+split large files. `tar cf - -C / $filename | split --bytes ${size}MB --verbose - $tmpfile.` My pure-Python implementation used the Tarfile library and I wrote a custom file-like object to split it--at least 50 lines. Both methods have pros/cons, but I had both implementations handy depending on the context I needed it in.

Another recent example was something I wrote to merge multiple video file "parts" into a single video using ffmpeg. It's 4 lines of shell script and would at least be 3x as long to write in Python.

The rule of thumb I have is if it's longer than 20 lines reevaluate if a shell scripting language is appropriate and I haven't really seen a need to change that in ~15 years I've used it.

Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to install and maintain other interpreters or runtimes on an embedded platform.

Even job management and parallelism are easier in Bash and GNU Parallel.

[1] https://www.tldp.org/LDP/abs/html/process-sub.html

The problem is that very few people know shell scripting well, but once you get to know it, it’s not that bad, in quite specific cases.

Always use Shellcheck, though.

I wanted to create a custom git command and started with Rust/libgit2, but found it was missing too much. Shell scripting proved to be the most natural, best supported approach (though it was nevertheless very painful).

1: https://github.com/git/git/blob/master/git-sh-setup.sh

2: https://github.com/ridiculousfish/git-prev-next

Eventually it motivated me to switch the language for shell scripting for the second time, and nowadays I recommend Haskell's Turtle (https://hackage.haskell.org/package/turtle-1.5.16/docs/Turtl...) to anyone who is interested in safer shell scripting and still likes a concise and terse syntax (and it's blazingly fast too).

If we understand "box" as "environment", in general, and not literally a physical or virtual machine...

Not really. E.g. see: every clean Ubuntu Docker container. They don't have Python, nor should have to include it by default.

And if you are going to do something like this:

  apt-get update && apt-get install -y stuff
  wget https://file.tgz
  tar xf file.tgz
  cd file
  make
  make install
  value="$(cat result | grep something)"
  mvn clean install -Dapp.value="$value"

I recently did the opposite of what everybody seems to be doing: moved an 800-line Python script, which you had to read carefully to understand what it was doing, to a 150-line Bash script that you only have to look at, because the important things to know is to see what other, external programs are being called. Just the apt-get stuff in Python was a mess with the poorly documented Python API for Apt, and some 50 lines to just do the same of an apt-get install... effectively wasted mental cycles.

(EDIT: formatting)

An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).

IMO, The inflection point where you should stop using shell is very low, though. The Google style guide for shell recommends that you should rewrite scripts once they are more than 100 lines, and I think that it probably too generous.

I think the main issue is: just because someone gets something to work on their system with their Python install, they think it works everywhere. And that is not at all true. There's a reason bin/sh|bash are so pervasive in administration, they are small, simple, well-understood, and resistant to the kind of bloat the Python chokes on.

python is basically this:

https://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B00...

perl still exists, and is a wonderful alternative to python for "shell scripts plus". The syntax should be at least vaguely familiar to anyone who has touched bash, awk, sed, etc.

bash has inherit_errexit to fix a longstanding bug, but even that's not enough.

Oil has shopt -s strict_errexit and more_errexit that fixes even more.

https://github.com/oilshell/oil/issues/476

I'm still looking for feedback and there's probably one more thing I want to change.

Although the fundamental problem is that errors (success/fail) and boolean values (true/false) are conflated in the status code, so it's a tough problem.

In Oil you will be encouraged to use expressions for booleans, not statements, e.g.

    ls /    # success/fail
    if (x > 0) { }  # true/false
    

    ls /
    if test $x -lt 0; then ...

But, in this case, because you're running it in a subcommand, you need to add pipefail.

    set -euo pipefail

Yes it can. You should use its exit status (or set -e). See http://man.openbsd.org/mktemp

Note that there is a nasty trap if you're using local to declare and set a variable:

    a () { local M=`ls /asdf` && echo y; }
    b () {       N=`ls /asdf` && echo y; }
    a ; b ;

For example, "TMPDIR=/dev/null mktemp -d" will reliably fail.

You shouldn't be validating it starts with "/tmp" though, because on quite a few systems, people set TMPDIR=/var/tmp.

You absolutely should check if mktemp exited nonzero, but if it exited 0, the directory should be safe to use. If you want to be really paranoid, you can check if the output is an empty string too after checking the exit code.

I think you're shying a little too far away from bash's globbing. Yeah, bash has footguns, but using clever workarounds can also be a recipe for creating confusing and error prone code in its own way.

  rm foo.[cC]

Mathematica and other software platforms use bash as universal installer for Mac and Linux. I have seen languages that compile to bash but use is limited based on the language.

From my experiment:

Googling "set euf" (without quotes) -- 838,000 results, first 6 results are all helpful and explain it in detail.

Googling "set o nounset" (without quotes) -- 38,700 results, entire first page seems helpful.

I dunno, looks like they're identically easy to learn the meaning of.

    x=(1 2)
    echo "${x[0]}"

    declare -A x
    x[hi]=20
    x[bye]=30
    echo "${x[hi]}"
    for i in "${!x[@]}"; do
        echo "k: $i, v: ${x[$i]}"
    done

Zsh is also really heavy, or it was last time I checked. Larger binary & slower startup time than bash, and lots of features I don’t use. Modules, calendar, tcp, ftp client ... that’s a lot of stuff in the shell that is arguably better put into separate executables.

You should use /usr/bin/env bash for the shebang. Some distros, such as nixos, don't have /bin/bash.

You need more quoting in the dirname line. cd "$(dirname "$0")" is what you need. The outer quotes are in case the directory has a space or special character in it, the inner ones are in case the script does.

That being said, you may also want $BASH_SOURCE rather than $0, but the reasons behind that are complicated. You may also want to support cases where the script is a symlink depending on what you're doing.

  #!/usr/bin/env bash

  bell=`tput bel`
  tock='Blastoff!'

  do_ring()
  {
    if [ "$1" ]; then # true
      echo -n $bell; sleep 0.1
      echo -n $bell; sleep 0.1
      echo -n $bell; sleep 0.1
      echo $tock
    else
      echo -n $bell; sleep 0.1
    fi
  }

  i=3
  while [ "$i" -ge "0" ];
  do
    if [ $i = 0 ]; then
      sleep 0.5
      echo ok
    else
      echo $i $bell
      sleep 0.5
    fi
    i=`expr $i - 1`
  done

  do_ring $1 # do something different if arg
  sleep 1
  echo neat.