Advanced Bash-Scripting Guide (2014)
tldp.org
tldp.org
Nobody who has ever used Bash has read the whole manual. It is actually good and useful, please read it. Yes, it is very big. But you will be using this shell for the rest of your life (even if you use Zsh, you will end up using other people's Bash scripts). Consider it an investment in your future.
...and now that I say that - try to restrict shell scripts to POSIX semantics (https://betterprogramming.pub/24-bashism-to-avoid-for-posix-...). POSIX is very simple and most shells can run it. If you need a Bashism, try Bash v3 first so it works on a Mac without Homebrew. Simple is better/portabler than complex. If you need something Bash doesn't provide, use any language that the majority of your team knows well.
If you want to test a specific version of Bash, there's containers @ https://hub.docker.com/_/bash (Macs use v3.2.57). Test POSIX scripts with `bash --posix`. Definitely use Shellcheck.
Bold claim and patently untrue because I have read the whole manual.
However:
1) Things you don't use often leak out of your brain at a rate that is absurd.
2) Different versions have different features and there's wide variations between versions on different platforms, you can't actually depend on a lot of bash stuff existing, so instead it's usually better to target POSIX sh
I once thought I’d solve this by taking notes, organizing a journal, and writing ... Well, it was a manual. That had the same problem as other manuals - too long, not gonna read.
Full circle.
Despite all this, my brain is still Swiss-cheese.
$ man bash | sed -n '/BUGS/,/^$/p'
BUGS
It's too big and too slow.
Except that doesn't disable all bashisms so you end up leaving in bashisms as a result, which is what caused all the issues for debian trying to migrate to dash
Here's the link to the latest official manual on GNU's site:
https://www.gnu.org/software/bash/manual/html_node/index.htm...
I have to admit, while i admire the idea of and understand the reasoning behind striving for POSIX-compatibility, i never couldn't fully get past viewing this as some kind of hypothetical exercise. The chance that any script i (or presumably most people who get this advice on pages like SO) will ever write will have to run on a 25 years old Tru64 machine or even a modern Mac, BSD, QNX, … is relatively small. Truth is (in my case), most of the scripts will never even run on anything but the system they were written on. As i said, i admire the idealism though.
First, we get rid of PATH= inside scripts. A lot of incompatibilites comes from /bin, not the shell itself. Treat executables like services, rewrite everything (reasonable) in pure shell.
The scenario I'm trying to prevent is "I want to cut a string in OSX but I wrote for a GNU executable and the syntax is slightly different". Most shell one liners can be rewritten as good quality portable code.
If you take just Debian, there are 5 or 6 shells available via apt. These should run wherever Debian runs (x86, arm, hurd, whatever).
You can't write a reasonable JavaScript that runs on JScript.exe (it still exists), Apple JXA, gjs and node.js. But you _could_, the language is stable enough, for long enough, that everyone knows how to feature detect the shit out of everything js related.
For the shell though, we don't even have a solid list of things that work or not in each shell, or differences across core utilities. Something like caniuse.com, but for UNIX shells, not browsers.
Having a page that runs on all browsers was once something impractical and idealistic. Not using tables and invisible 1x1 pixels for layouts was once idealistic. We fixed it though. Mostly.
But the reason to use POSIX scripting isn't portability, it's simplicity. Simple scripts are just easier to reason about, easier to maintain, and are less prone to bugs. The code might be a little more verbose, but its simplicity still reduces the cost of complexity.
Most people don't realize this until they've been writing scripts for 15 years and realize they can do almost everything with POSIX and that it makes their life easier in the long run. But in some cases what I want to do would really benefit from Bash, so I use Bash then. It's not a hard and fast rule; I switch based on what would work best for that case. It's about balance, compromise, and being flexible.
Much, much better is the Bash Hacker's Wiki:
https://wiki.bash-hackers.org/
And Wooledge's Wiki:
We never left the hell caused by the war between shells to provide interactive features. They were all feature creeped for terminal usage and the scripting API was left to rot.
I get the people that say we should use python or something else instead. It won't work though if that language is not capable of replacing the shell _as a build dependency_. As long as we need the shell to build stuff, we're stuck to it.
Dockerfiles, GitHub Actions YAMLs, all these tools accept shell input. This is shiny new software that could have picked any language for their proprietary DSLs, but the shell was chosen.
What the shell needs is what happened to JavaScript around the 2000s. Different interpreters started to align, standards were written, reusable libraries started to pop up everywhere, and now it runs in our brains. Before all of that, JavaScript was merely a cute toy to animate pages.
There’s a place reserved in hell for whoever thought it was a good idea to design a build system around shell code in yaml files.
I’m in the process of moving thousands of lines of such code into standalone scripts that then get called from the yaml files. Keep the yaml as minimal as possible. Then I can shellcheck the shell code or easily port bits to other languages as needed.
I saw it first here: https://github.com/shellfire-dev/core/blob/master/variable/a...
For many cases, this is even better than bash native arrays. You can pass these as arguments and preserve the structure, while the bash ones will autoexpand if you try that.
I ended up cooking my own array implementation later, but it is a bit more unorthodox (I create extra vars with numeric pointers to handle the values). Let me know if you want to see it, it's not complete but it works.
I'm not very fond of YAML either, I feel your pain :(
What happens if the delimiter appears among the values you want to store? This is a pretty big disclaimer:
> Do not use arrays with file paths if there's any chance they could contain \r.
var lorem = text_create "Hello"
var foo = array_create
array_push $foo $lorem
`var`, `array_push` and `array_create` are shell functions working together. They create pointers and return just integers. You don't even have to quote the variable.
The code above should create the following variables:
_123="Hello" _124=1 _124_0=_123 foo=_124 lorem=_123
This way, I can even implement more complex data structures.
The numbers start from zero and the program will stop if the global variable counter overflows. My current implementation also has a primitive GC (ARC-based), which could be used to avoid that (but I didn't, yet).
Despite all of this runtime processing and jerry rigging, I have some evidence that it could be faster than typical shell scripts. One single subshell or path lookup is often more expensive than these fake pointer arrays. These strucutres have zero subshells and run on PATH='', just with builtins. The subshell only becomes faster if the volume of data is larger than a few kilobytes (when the cost of forking and exec'in is worth the payoff).
I know this is far from ideal. I should try to put these ideas in a new interpreter, not hack them on existing ones (JS calls them polyfills, maybe I should do that). If I did another interpreter though, it would end up like the fish shell (nice, fast, doesn't compile the kernel and no one uses it).
Maybe I should probably stick to bash, ksh and zsh that offers the most complete set of features. Sounds like "best viewed in internet explorer" though.
You don't need to escape new lines if you end them with an operator:
test 1 = 1 &&
test a = b ||
exit 122
Works with all logical operators, ampersand and pipe. By the way, the ampersand is often expressed with a new line after it, but it doesn't require it:
this_in_background & this_in_foreground
This is the same as writing:
this_in_background &
this_in_foreground
tail -n $lines messages > mesg.temp
mv mesg.temp messages
Which kind of reminds me of that product running on a pseudo-distributed NoSQL database built out of text files with "record per line" structure and sort of a compacting garbage collector for these files to remove dead records. And there's just absolutely nothing that can go wrong with that.
I think you're missing the forest for the trees.
Bash is installed by default because Bash is specified in an international standard dubbed the Portable Operating System Interface (POSIX). Your bash scripts are not backwards compatible. Your bash scripts are compatible, and portable, and standardized, because that's their point, and the whole point of POSIX.
No one is held back by bash or POSIX. You are free to use anything that suits your fancy. Feel free to whip out scripts in Python or Perl or Ruby. Some people do, and last time I checked some operating systems like macOS shipped their interpreters by default. Odds are you are free to easily install those interpreters in those who don't.
random high result from googling "bashism": https://betterprogramming.pub/24-bashism-to-avoid-for-posix-...
The point is that there are many improvements that could be made to bash which would retain the good properties but would get rid of the worst footguns and suckiness. Quite a number of these (though not all) have been available for decades in the form of zsh.
Also, as pointed out already in another comment, bash is not in POSIX at all. POSIX is very much a "lowest common denominator"-kind of specification which moves at geological speeds. Even dash and busybox sh implement some non-POSIX features.
The shell's power & it's footguns are one in the same. "Fix" them, and becomes a general-purpose language--a jack of all trades & master of none. CS professors will love it, but doing anything meaningful in the shell will look more like a one-page python script instead of the l337 one-liners the Jersey boys graced us with.
I want to see someone try "bash but we really embrace the insanity."
For instance, make recursively affecting all files in subdirectories the default? Or have a global option where any prompt is automatically answered with yes? Maybe some heuristics to detect if the user wants unquoted variables expanded?
That's what zsh did; $var is always the value of $var. Want field splitting? Use $=var. Want globbing? Use $~var.
I think bash is stuck in 1988 and is a poor clone of ksh. I think it's highly unfortunate it became the "de-facto 'modern' standard".
Perhaps you're right. Perhaps. However, the most important consequence of bash creation and dissemination is that there is a de-facto standard.
So until we don't have another de-facto standard hundreds of thousands of typical shell users are happy with bash. With all its issues, weaknesses and disadvantages...
I also think the value of being the de-facto standard is somewhat overrated (as is the value of portability by the way); many scripts only run on one platform, one you control. zsh is not hard to install.
Bash is just too damn fast to whip things up in (even at the risk of endless footguns). Build up a pipeline of commands, translate into script, done. It's powerful and can scale if common conventions and good practices are used.
For easier debugging I always add a simple flag along the lines of: if $1 = -v then set -x; shift
Obviously shellscripts are a terrible or highly suspect idea when production-grade transactional integrity is critical, but in many cases the swiftness of shell development outweighs the cons when the goal is to Get Things Done.
The shell is a pretty sweet interface.
When I was done I looked over it and thought: this is a nice piece of code right now. It's well documented. It's self contained. It passes my tests. I could maintain this code.
But it made use of quite a few bash'isms: arrays and so forth. I wasn't confident in the ability of my colleagues to keep it in good state.
So I rewrote it in Python. The Python version ended up being a little longer, but that was mostly due to formatting difference. The Python version in addition I could reformat with black and isort, I added type annotations so I could test it with mypy.
(I had formatting the shell version with shfmt and checked it with shellcheck, but still.)
The Python version is much easier to read, and I think will be more maintainable.
I consider myself competent in both languages so it's a courtesy to my colleagues if nothing else to ship the Python version.
But now that I can expect python3 to be available on the systems I care about, and with the fairly comfortable ergonomics of subprocess.run, I find myself using Python in places I would've previously used bash.
A good heuristic is the moment a bash script needs to use curl + jq, it's time for python. Especially on macOS it's nice having things like plistlib so I can avoid `defaults` and `PlistBuddy`.
Python sucks for building processing pipelines.
How many lines of Python code would you need to implement this?
paste -d \\n <(foo …) <(bar …) | while read -r f && read -r b; do
baz “$f” “$b”
done
replace 200+ bash script with multiple subroutines in python? yes
from subprocess import Popen, PIPE, run
with Popen('foo', stdout=PIPE) as foo, Popen('bar', stdout=PIPE) as bar:
for f, b in zip(foo.stdout, bar.stdout):
run(['baz', f, b])
Python is pretty safe, but your code would have to work with both 2 & 3 and not pull in any external dependencies. And recommended practices have changed so drastically on how to install Python over the years that a random dev could have theirs in a fairly weird state.
Perl would work, but hacked-together Perl has the similar drawbacks to hacked-together Bash, especially when it's me doing the coding.
Ruby has problems similar to Python.
Make & awk have problems similar to Perl & Bash.
Therefore my latest rust package contains bash scripts.
Edit: it's a genuine question, I have no idea what everyone else is seeing.
<<EOF (expands variables etc)
<<'EOF' (doesn't expand variables etc)
<<-EOF (trims leading tabs)
Am I missing any more of the tricks?
sed 's/^ *//' <<EOF
Which I'd prefer anyways because it's less cryptic. I try to stay away from the obscure, rarely used notations because I'll always have to look it up in StackOverflow anytime I come back to it. These can be confusingly sharp edges.
while read -r LINE
do printf %s "$LINE"
done <<FOO
foo
bar
baz
FOO
This is faster than using cat (one less path lookup, one less fork, one less exec, read is builtin), works on all shells.
There is a <<-'EOF' as well (trims left tabs and does not expand vars).
Another cool trick that only bash and zsh have is `printf -v NAME`, which prints to a variable instead of stdout. You can use it to avoid subshells (one less fork for each subshell avoided).
A language that mixes code and data as the same variable type, with only a single missing quote needed to turn any variable into a full RCE. On top of that, add all the obscure syntax, stringly typed data for everything and with whatever measly error handling it actually does support being defaulted to off. Guess what you get.