Hacker News Reader

Comment by koolba | Hacker News Reader

The first two lines of the script are already wrong;

    #!/bin/bash

Should be:

    #!/usr/bin/env bash
    set -euo pipefail

That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

View on HN

I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105

It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.

Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.

No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.

If I'm writing a bash script I run manually and it's more than a handful of functions I agree. I'm there to babysit it and its probably complicated.

If the bash script is ran on thousands of containers where its not possible to babysit. My number one job is to stop immediately when an error happens and surface that error to any monitoring system.

I agree about that distinction, but still, won’t that error need to be formatted? Is it safe to rely on logging picking up on the error or does the ”simple script” solution imply that there is monitoring for the script exit code?

If I may add a few critiques...

  # Make sure no one else is using rsync
  pro_on=$(ps aux | grep -c rsync)

A better way to do that is with the flock utility.

  (
    flock -n 9 || exit 1 # Critical section-allow only one process.

    ...single thread shell script

  ) 9> ~/.empty_lock_file

Note that the flock utility is specific to Linux, but POSIX mkdir() is atomic and could be more portable.

  "${SOURCES[@]}"

POSIX shells do not support arrays. Iterating with read over a here document is more portable.

  minutes=$(($minutes - 1))

POSIX is specific that the $ prefix on a variable name can be omitted in an arithmetic expression.

  ECHO="/bin/echo"

Many shell scripts never use echo, and this is a good idea. 'NEVER use echo like this. According to POSIX, echo has unspecified behavior if any of its arguments contain "\" or if its first argument is "-n".' http://www.etalabs.net/sh_tricks.html

Perhaps use this instead, in a subshell to avoid stomping on variables:

  myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )

#!/usr/bin/env bash and set -u are always good ideas.

There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.

Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted.

When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like

  [ "${DEBUG:-0}" = "1" ] && set -x

> Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted.

Really? In this script's "ps aux | grep -c rsync" for example, if "ps" fails, you'll just get 0 without the grep failing.

(Speaking of that line: chasil's completely right that it's much better to use "flock" than "ps" for locking...)

> There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.

You don't let go of -e for that.

  dont_mind_failure || true
  important_process

add 'true' specifically if you must.

There are scripts where you don't want any failure to kill the script. You would have to wrap nearly every line in that kind of exception handling, and if you miss one, the script fails. It is much simpler to omit -e in that case

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues.

Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

> "#!/bin/bash" works fine

It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin

Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.

does it matter? MacOS is using some ancient ass 3.x bash from 2007. The location of bash is the least of your worries when it comes to portability. There's no guarantee any of the commands in your script (a) exist on the system or (b) work with the same options and flags that the script uses. I don't even know how many "netcat" commands I've seen in the wild. You could be running in some BusyBox or pared down Docker container. Blindly running a script that hasn't been deliberately crafted to work on your system is just asking for trouble.

You call 2007 ancient when bash has been around for over 30 years?

What exactly has changed in the last 15 years?

Agreed on fail on error, but the first one is needlessly pedantic.

Find me a single Linux distro where bash, if installed, is not available in /bin

I'd argue it's easier to find a distro without /usr/bin/env.

NixOS

As a system eng for almost 15 years I've spent a good number of years handling deployments, dependancies, environment evaluations and all the little minutiae that's required to run large complex distributed systems at scale. This is fine to ignore if it's your personal box and no one else is working on it.

However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' prod setups where the system bash and the deployed environment may contain different bash.

The different bash is the one that was tested in the test env with the automation. There may even be multiple different versions of an interpreter on the system with multiple app environments running.

This was a pretty common way to package apps before easy access to containers and container managers.

This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.

I was a sysadmin on SunOS, HP/UX and BSD systems back in the day and have been maintaining various Linux-based systems in the past few decades as well.

We're not talking about Google data center here, it's just someone's shell script.

> This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.

We don't have industry best practices for shell scripts, no matter what consultants / HN commenters with strong opinions say. You can see in this thread that folks disagree about what the best practices are. It's worth paying attention to folks' rationale for their opinions—I learned some caveats about "-e" from following the links here. But the talk about "table stakes" and "industry best practices" is (extended bleep). Those don't exist.

Different projects/companies may have their own best practice guides that make sense in their environment. senko mentioned Google data centers as a place where it might make sense to be more rigorous. Google's guide says to use "#!/bin/bash". [1] If that doesn't work in your environment, fine, but that doesn't make them wrong.

Shell is a surprisingly and unnecessarily difficult language to write correctly. To the extent there is a best practice on it, I think it's "use a better language for anything that might become large or important". The Google style guide I linked says more or less the same thing near the beginning. The subtleties discussed in the rest give you a taste of why...

[1] https://google.github.io/styleguide/shellguide.html

1. bash runs on more than just linux

2. Many non-LSB distros

The article talks about Linux.

There are more than Linux in the real world.

On my laptop (admittedly, it's running Mac OS, not a linux distro), /bin/bash is version 3.2.57 released in 2007, while '/usr/bin/env bash' invokes bash 5.1.8, released in 2020.

Not using “env” isn’t “wrong”. It depends how many platforms they want to supply

Bet yes pipefail and nounset should definitely be set.

Support*

Reply on news.ycombinator.com